[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:33:17 dist-test-slave-t7tk krb5kdc[10639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:33:17 dist-test-slave-t7tk krb5kdc[10639](info): set up 2 sockets
Sep 26 12:33:17 dist-test-slave-t7tk krb5kdc[10639](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 12:33:18.368144 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:33:18 dist-test-slave-t7tk krb5kdc[10639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:33:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:33:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:33:18.493636 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:35825
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:39355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:35825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:18.606921 10652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:18.607224 10652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:18.607283 10652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:18.611472 10652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:18.611553 10652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:18.611582 10652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:18.611605 10652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:18.611634 10652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:33:18.617305 10652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:39355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:35825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:35825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:18.618379 10652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:18.619415 10652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:18.628281 10658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:18.628389 10652 server_base.cc:1034] running on GCE node
W20260926 12:33:18.628309 10660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:18.628309 10657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:18.629132 10652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:18.630300 10652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:18.631477 10652 hybrid_clock.cc:648] HybridClock initialized: now 1790425998631460 us; error 30 us; skew 500 ppm
Sep 26 12:33:18 dist-test-slave-t7tk krb5kdc[10639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425998, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:18.634462 10652 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:33:18.635469 10652 webserver.cc:460] Webserver started at http://127.10.98.62:35233/ using document root <none> and password file <none>
I20260926 12:33:18.636032 10652 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:18.636122 10652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:18.636379 10652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:18.638748 10652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "d4d589190ea94fcc9af2594ecc8cdf15"
format_stamp: "Formatted at 2026-09-26 12:33:18 on dist-test-slave-t7tk"
server_key: "2604fcdccd0e1af35705f32118240eb0"
server_key_iv: "13933ce7d092c11dd4e777a4caa32970"
server_key_version: "encryptionkey@0"
I20260926 12:33:18.639429 10652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "d4d589190ea94fcc9af2594ecc8cdf15"
format_stamp: "Formatted at 2026-09-26 12:33:18 on dist-test-slave-t7tk"
server_key: "2604fcdccd0e1af35705f32118240eb0"
server_key_iv: "13933ce7d092c11dd4e777a4caa32970"
server_key_version: "encryptionkey@0"
I20260926 12:33:18.643632 10652 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:33:18.646291 10667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:18.647650 10652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:33:18.647779 10652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "d4d589190ea94fcc9af2594ecc8cdf15"
format_stamp: "Formatted at 2026-09-26 12:33:18 on dist-test-slave-t7tk"
server_key: "2604fcdccd0e1af35705f32118240eb0"
server_key_iv: "13933ce7d092c11dd4e777a4caa32970"
server_key_version: "encryptionkey@0"
I20260926 12:33:18.647959 10652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:18.663230 10652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:18.666798 10652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:18.666998 10652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:18.674153 10652 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:35825
I20260926 12:33:18.674170 10719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:35825 every 8 connection(s)
I20260926 12:33:18.675132 10652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:33:18.678004 10720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:18.680436 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 10652
I20260926 12:33:18.680574 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:33:18.682000 10632 external_mini_cluster.cc:1428] Setting key 0c2ed6f6e72430d97d2fd90b320e249a
I20260926 12:33:18.683705 10720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15: Bootstrap starting.
I20260926 12:33:18.686173 10720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:18.686944 10720 log.cc:824] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:18.688910 10720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15: No bootstrap required, opened a new log
I20260926 12:33:18.693081 10720 raft_consensus.cc:348] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d589190ea94fcc9af2594ecc8cdf15" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35825 } }
I20260926 12:33:18.693403 10720 raft_consensus.cc:374] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:18.693467 10720 raft_consensus.cc:729] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4d589190ea94fcc9af2594ecc8cdf15, State: Initialized, Role: FOLLOWER
I20260926 12:33:18.693970 10720 consensus_queue.cc:260] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d589190ea94fcc9af2594ecc8cdf15" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35825 } }
I20260926 12:33:18.694126 10720 raft_consensus.cc:388] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:33:18.694187 10720 raft_consensus.cc:482] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:33:18.694278 10720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:18.695240 10720 raft_consensus.cc:504] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d589190ea94fcc9af2594ecc8cdf15" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35825 } }
I20260926 12:33:18.695564 10720 leader_election.cc:302] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d4d589190ea94fcc9af2594ecc8cdf15; no voters:
I20260926 12:33:18.695863 10720 leader_election.cc:288] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:33:18.695952 10725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:18.696138 10725 raft_consensus.cc:686] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15 [term 1 LEADER]: Becoming Leader. State: Replica: d4d589190ea94fcc9af2594ecc8cdf15, State: Running, Role: LEADER
I20260926 12:33:18.696434 10725 consensus_queue.cc:237] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d589190ea94fcc9af2594ecc8cdf15" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35825 } }
I20260926 12:33:18.697188 10720 sys_catalog.cc:564] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:18.697616 10725 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d4d589190ea94fcc9af2594ecc8cdf15. Latest consensus state: current_term: 1 leader_uuid: "d4d589190ea94fcc9af2594ecc8cdf15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d589190ea94fcc9af2594ecc8cdf15" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35825 } } }
I20260926 12:33:18.697731 10725 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:18.698159 10729 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:33:18.700387 10726 sys_catalog.cc:455] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d4d589190ea94fcc9af2594ecc8cdf15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d4d589190ea94fcc9af2594ecc8cdf15" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35825 } } }
I20260926 12:33:18.700490 10726 sys_catalog.cc:458] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15 [sys.catalog]: This master's current role is: LEADER
Sep 26 12:33:18 dist-test-slave-t7tk krb5kdc[10639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:18.702147 10729 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:33:18.707551 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:18.685179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.10.98.62:35825)
0926 12:33:18.685692 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:18.685711 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:18.686706 (+ 995us) server_negotiation.cc:407] Connection header received
0926 12:33:18.690142 (+ 3436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:18.690164 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:18.690563 (+ 399us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:18.690916 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:18.693024 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:18.694473 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:18.695771 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:18.696449 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:18.696603 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:18.701866 (+ 5263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:18.701872 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:18.701889 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:18.701922 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:18.703717 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:18.704341 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:18.704347 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:18.704352 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:18.704494 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:18.704891 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:18.704895 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:18.704897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:18.705442 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:18.705650 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:18.706302 (+ 652us) server_negotiation.cc:299] Negotiation successful
0926 12:33:18.706598 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":160,"threads_started":1}
I20260926 12:33:18.709342 10729 catalog_manager.cc:1347] Generated new cluster ID: 9b366d7bd70d46dd92d7a5d54fad79a6
I20260926 12:33:18.709398 10729 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:33:18.740063 10729 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:33:18.741036 10729 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:33:18.754171 10729 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d4d589190ea94fcc9af2594ecc8cdf15: Generated new TSK 0
I20260926 12:33:18.754971 10729 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
I20260926 12:33:18.789645 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:35825
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:39355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:18.921542 10746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:18.921805 10746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:18.921849 10746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:18.925838 10746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:18.925904 10746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:18.925962 10746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:33:18.931329 10746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:39355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:35825
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:18.932350 10746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:18.933351 10746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:18.942992 10751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:18.943013 10754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:18.943226 10746 server_base.cc:1034] running on GCE node
W20260926 12:33:18.943037 10752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:18.943728 10746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:18.944370 10746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:18.945523 10746 hybrid_clock.cc:648] HybridClock initialized: now 1790425998945497 us; error 38 us; skew 500 ppm
Sep 26 12:33:18 dist-test-slave-t7tk krb5kdc[10639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425998, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:18.948344 10746 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:33:18.949338 10746 webserver.cc:460] Webserver started at http://127.10.98.1:46459/ using document root <none> and password file <none>
I20260926 12:33:18.949883 10746 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:18.949944 10746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:18.950141 10746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:18.952306 10746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "7a62b2ae2e494e72bbca4e720c44f5bf"
format_stamp: "Formatted at 2026-09-26 12:33:18 on dist-test-slave-t7tk"
server_key: "9dbc53d341792a682cc834fe94fd2cfc"
server_key_iv: "78d7297dcdfeaced1b7facdd5f3b1e73"
server_key_version: "encryptionkey@0"
I20260926 12:33:18.952862 10746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "7a62b2ae2e494e72bbca4e720c44f5bf"
format_stamp: "Formatted at 2026-09-26 12:33:18 on dist-test-slave-t7tk"
server_key: "9dbc53d341792a682cc834fe94fd2cfc"
server_key_iv: "78d7297dcdfeaced1b7facdd5f3b1e73"
server_key_version: "encryptionkey@0"
I20260926 12:33:18.956701 10746 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260926 12:33:18.959271 10761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:18.960582 10746 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:33:18.960701 10746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "7a62b2ae2e494e72bbca4e720c44f5bf"
format_stamp: "Formatted at 2026-09-26 12:33:18 on dist-test-slave-t7tk"
server_key: "9dbc53d341792a682cc834fe94fd2cfc"
server_key_iv: "78d7297dcdfeaced1b7facdd5f3b1e73"
server_key_version: "encryptionkey@0"
I20260926 12:33:18.960820 10746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:18.973209 10746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:18.976153 10746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:18.976334 10746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:18.976897 10746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:18.977732 10746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:18.977789 10746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:18.977842 10746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:18.977866 10746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:18.986003 10746 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:43739
I20260926 12:33:18.986040 10874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:43739 every 8 connection(s)
I20260926 12:33:18.986876 10746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
Sep 26 12:33:18 dist-test-slave-t7tk krb5kdc[10639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425998, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:18.996784 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 10746
I20260926 12:33:18.996892 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:33:18.997211 10632 external_mini_cluster.cc:1428] Setting key b79679f96b53004206e21ed4bed706d6
I20260926 12:33:19.001394 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:18.988794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:49179 (local address 127.10.98.62:35825)
0926 12:33:18.988958 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:18.988962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:18.989848 (+ 886us) server_negotiation.cc:407] Connection header received
0926 12:33:18.991171 (+ 1323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:18.991174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:18.991222 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:18.991300 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:18.992537 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:18.993091 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:18.994358 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:18.994592 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:18.994683 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:18.997345 (+ 2662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:18.997350 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:18.997354 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:18.997389 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:18.999138 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:18.999727 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:18.999730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:18.999732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:18.999821 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:19.000231 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:19.000235 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:19.000237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:19.000424 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:19.000517 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:19.001075 (+ 558us) server_negotiation.cc:299] Negotiation successful
0926 12:33:19.001232 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 12:33:19.002233 10877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:18.989212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:35825 (local address 127.10.98.1:49179)
0926 12:33:18.989681 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:18.989727 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:18.990927 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:18.991457 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:18.991463 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:18.991819 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:18.992370 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:18.992384 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:18.993265 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:18.993268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:18.994242 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:18.994252 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:18.994745 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:18.994749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:18.994942 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:18.995498 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:18.995515 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:18.997084 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:18.999276 (+ 2192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:18.999284 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:18.999298 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:18.999626 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:18.999935 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:18.999938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:18.999939 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:19.000120 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:19.000537 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:19.000543 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:19.000824 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 12:33:19.001006 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 12:33:19.001373 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":105,"threads_started":1}
I20260926 12:33:19.003561 10875 heartbeater.cc:347] Connected to a master server at 127.10.98.62:35825
I20260926 12:33:19.003814 10875 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:19.004391 10875 heartbeater.cc:511] Master 127.10.98.62:35825 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.98.2@KRBTEST.COM; defaulting to no policy
I20260926 12:33:19.006042 10684 ts_manager.cc:194] Registered new tserver with Master: 7a62b2ae2e494e72bbca4e720c44f5bf (127.10.98.1:43739)
I20260926 12:33:19.007333 10684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:49179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
I20260926 12:33:19.023597 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:35825
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:39355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:19.132946 10880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:19.133262 10880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:19.133307 10880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:19.137318 10880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:19.137385 10880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:19.137444 10880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:33:19.142947 10880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:39355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:35825
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:19.144021 10880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:19.145031 10880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:19.155059 10888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:19.155063 10885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:19.155063 10886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:19.155354 10880 server_base.cc:1034] running on GCE node
I20260926 12:33:19.155843 10880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:19.156498 10880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:19.157656 10880 hybrid_clock.cc:648] HybridClock initialized: now 1790425999157624 us; error 46 us; skew 500 ppm
Sep 26 12:33:19 dist-test-slave-t7tk krb5kdc[10639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425999, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:19.160359 10880 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:33:19.161396 10880 webserver.cc:460] Webserver started at http://127.10.98.2:33515/ using document root <none> and password file <none>
I20260926 12:33:19.161955 10880 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:19.162027 10880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:19.162225 10880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:19.164328 10880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "9444ad38d1204f1ab527f286e97aa2e1"
format_stamp: "Formatted at 2026-09-26 12:33:19 on dist-test-slave-t7tk"
server_key: "a8a92f031d456d72a2125d83b53a8a4f"
server_key_iv: "d98efcce29442fdcb839466cd6b56e25"
server_key_version: "encryptionkey@0"
I20260926 12:33:19.164883 10880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "9444ad38d1204f1ab527f286e97aa2e1"
format_stamp: "Formatted at 2026-09-26 12:33:19 on dist-test-slave-t7tk"
server_key: "a8a92f031d456d72a2125d83b53a8a4f"
server_key_iv: "d98efcce29442fdcb839466cd6b56e25"
server_key_version: "encryptionkey@0"
I20260926 12:33:19.168772 10880 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.000s
I20260926 12:33:19.171725 10895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:19.173240 10880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:33:19.173372 10880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "9444ad38d1204f1ab527f286e97aa2e1"
format_stamp: "Formatted at 2026-09-26 12:33:19 on dist-test-slave-t7tk"
server_key: "a8a92f031d456d72a2125d83b53a8a4f"
server_key_iv: "d98efcce29442fdcb839466cd6b56e25"
server_key_version: "encryptionkey@0"
I20260926 12:33:19.173499 10880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:19.189015 10880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:19.192137 10880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:19.192281 10880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:19.192813 10880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:19.193648 10880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:19.193694 10880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:19.193733 10880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:19.193747 10880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:19.202639 10880 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:38851
I20260926 12:33:19.202687 11008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:38851 every 8 connection(s)
I20260926 12:33:19.203603 10880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:33:19.209741 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 10880
I20260926 12:33:19.209834 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:33:19.210052 10632 external_mini_cluster.cc:1428] Setting key 82830529376f4758883877a99f10a065
Sep 26 12:33:19 dist-test-slave-t7tk krb5kdc[10639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425999, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:19.217514 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:19.205371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:46467 (local address 127.10.98.62:35825)
0926 12:33:19.205509 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:19.205513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:19.206672 (+ 1159us) server_negotiation.cc:407] Connection header received
0926 12:33:19.207519 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:19.207523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:19.207578 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:19.207657 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:19.208838 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:19.209418 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:19.210627 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:19.210927 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:19.211029 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:19.213685 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:19.213689 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:19.213693 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:19.213722 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:19.215165 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:19.215823 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:19.215826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:19.215827 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:19.215907 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:19.216273 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:19.216276 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:19.216277 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:19.216456 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:19.216529 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:19.217272 (+ 743us) server_negotiation.cc:299] Negotiation successful
0926 12:33:19.217391 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:33:19.218741 11011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:19.205841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:35825 (local address 127.10.98.2:46467)
0926 12:33:19.206502 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:19.206544 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:19.207314 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:19.207839 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:19.207847 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:19.208202 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:19.208694 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:19.208709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:19.209536 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:19.209539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:19.210502 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:19.210513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:19.211114 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:19.211118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:19.211325 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:19.212018 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:19.212035 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:19.213493 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:19.215323 (+ 1830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:19.215329 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:19.215346 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:19.215718 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:19.215994 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:19.215997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:19.215999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:19.216183 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:19.216550 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:19.216558 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:19.216917 (+ 359us) client_negotiation.cc:769] Sending connection context
0926 12:33:19.217218 (+ 301us) client_negotiation.cc:240] Negotiation successful
0926 12:33:19.217599 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.10.98.3@KRBTEST.COM; defaulting to no policy
I20260926 12:33:19.220006 11009 heartbeater.cc:347] Connected to a master server at 127.10.98.62:35825
I20260926 12:33:19.220275 11009 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:19.220893 11009 heartbeater.cc:511] Master 127.10.98.62:35825 requested a full tablet report, sending...
I20260926 12:33:19.222162 10684 ts_manager.cc:194] Registered new tserver with Master: 9444ad38d1204f1ab527f286e97aa2e1 (127.10.98.2:38851)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.3@KRBTEST.COM" created.
I20260926 12:33:19.222843 10684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:46467
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
I20260926 12:33:19.240761 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:35825
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:39355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:19.351352 11014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:19.351615 11014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:19.351668 11014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:19.355782 11014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:19.355854 11014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:19.355913 11014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:33:19.361703 11014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:39355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:35825
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11014
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:19.362829 11014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:19.363888 11014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:19.374295 11019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:19.374406 11014 server_base.cc:1034] running on GCE node
W20260926 12:33:19.374346 11020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:19.374349 11022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:19.375000 11014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:19.375679 11014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:19.376857 11014 hybrid_clock.cc:648] HybridClock initialized: now 1790425999376812 us; error 61 us; skew 500 ppm
Sep 26 12:33:19 dist-test-slave-t7tk krb5kdc[10639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425999, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:19.379727 11014 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:33:19.380769 11014 webserver.cc:460] Webserver started at http://127.10.98.3:45143/ using document root <none> and password file <none>
I20260926 12:33:19.381356 11014 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:19.381426 11014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:19.381640 11014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:19.383723 11014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "a45eb990de874adf82bde86e2bb8e3b9"
format_stamp: "Formatted at 2026-09-26 12:33:19 on dist-test-slave-t7tk"
server_key: "4d4f92deeb6d816756adc575c255c5d8"
server_key_iv: "435b714b095925cc72bd306444f70f3f"
server_key_version: "encryptionkey@0"
I20260926 12:33:19.384290 11014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "a45eb990de874adf82bde86e2bb8e3b9"
format_stamp: "Formatted at 2026-09-26 12:33:19 on dist-test-slave-t7tk"
server_key: "4d4f92deeb6d816756adc575c255c5d8"
server_key_iv: "435b714b095925cc72bd306444f70f3f"
server_key_version: "encryptionkey@0"
I20260926 12:33:19.388175 11014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 12:33:19.390765 11029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:19.391948 11014 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:33:19.392062 11014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "a45eb990de874adf82bde86e2bb8e3b9"
format_stamp: "Formatted at 2026-09-26 12:33:19 on dist-test-slave-t7tk"
server_key: "4d4f92deeb6d816756adc575c255c5d8"
server_key_iv: "435b714b095925cc72bd306444f70f3f"
server_key_version: "encryptionkey@0"
I20260926 12:33:19.392165 11014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:19.435784 11014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:19.439124 11014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:19.439309 11014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:19.439916 11014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:19.440783 11014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:19.440831 11014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:19.440884 11014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:19.440912 11014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:19.449808 11014 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:32939
I20260926 12:33:19.449898 11142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:32939 every 8 connection(s)
I20260926 12:33:19.450762 11014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:33:19.457501 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 11014
I20260926 12:33:19.457621 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:33:19.457863 10632 external_mini_cluster.cc:1428] Setting key 6765b8f4c147ab4d7c87ef5fe87feff2
Sep 26 12:33:19 dist-test-slave-t7tk krb5kdc[10639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425999, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:19.464915 10722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:19.452772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:57551 (local address 127.10.98.62:35825)
0926 12:33:19.452916 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:19.452921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:19.453945 (+ 1024us) server_negotiation.cc:407] Connection header received
0926 12:33:19.455245 (+ 1300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:19.455248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:19.455297 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:19.455370 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:19.456582 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:19.457185 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:19.458384 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:19.458657 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:19.458765 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:19.461449 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:19.461453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:19.461455 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:19.461487 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:19.462854 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:19.463342 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:19.463346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:19.463347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:19.463432 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:19.463794 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:19.463797 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:19.463798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:19.464067 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:19.464159 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:19.464659 (+ 500us) server_negotiation.cc:299] Negotiation successful
0926 12:33:19.464792 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:33:19.465780 11145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:19.453314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:35825 (local address 127.10.98.3:57551)
0926 12:33:19.453791 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:19.453837 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:19.455033 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:19.455529 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:19.455537 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:19.455934 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:19.456437 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:19.456451 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:19.457289 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:19.457292 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:19.458262 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:19.458272 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:19.458815 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:19.458820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:19.459011 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:19.459614 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:19.459630 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:19.461224 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:19.462964 (+ 1740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:19.462969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:19.462982 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:19.463240 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:19.463528 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:19.463531 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:19.463532 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:19.463685 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:19.464170 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:19.464175 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:19.464427 (+ 252us) client_negotiation.cc:769] Sending connection context
0926 12:33:19.464589 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 12:33:19.464896 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":107,"threads_started":1}
I20260926 12:33:19.466871 11143 heartbeater.cc:347] Connected to a master server at 127.10.98.62:35825
I20260926 12:33:19.467144 11143 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:19.467696 11143 heartbeater.cc:511] Master 127.10.98.62:35825 requested a full tablet report, sending...
I20260926 12:33:19.468778 10684 ts_manager.cc:194] Registered new tserver with Master: a45eb990de874adf82bde86e2bb8e3b9 (127.10.98.3:32939)
I20260926 12:33:19.469381 10684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:57551
I20260926 12:33:19.472483 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 12:33:19 dist-test-slave-t7tk krb5kdc[10639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425999, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 12:33:19 dist-test-slave-t7tk krb5kdc[10639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425999, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.98.1@KRBTEST.COM
I20260926 12:33:19.499080 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:19.488938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58480 (local address 127.10.98.1:43739)
0926 12:33:19.489267 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:19.489284 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:19.489354 (+ 70us) server_negotiation.cc:407] Connection header received
0926 12:33:19.489498 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:19.489504 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:19.489751 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:19.489887 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:19.490597 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:19.491366 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:19.492195 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:19.492567 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:19.492672 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:19.495155 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:19.495165 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:19.495180 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:19.495226 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:19.496939 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:19.497549 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:19.497555 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:19.497556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:19.497661 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:19.498017 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:19.498019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:19.498020 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:19.498279 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:19.498517 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:19.498713 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 12:33:19.498886 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":86,"threads_started":1}
W20260926 12:33:19.500114 10789 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58480
Sep 26 12:33:19 dist-test-slave-t7tk krb5kdc[10639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 12:33:19 dist-test-slave-t7tk krb5kdc[10639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790425999, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.1@KRBTEST.COM
I20260926 12:33:19.526095 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:19.517591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58490 (local address 127.10.98.1:43739)
0926 12:33:19.517778 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:19.517783 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:19.517879 (+ 96us) server_negotiation.cc:407] Connection header received
0926 12:33:19.518094 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:19.518097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:19.518145 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:19.518228 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:19.519030 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:19.519549 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:19.520392 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:19.520631 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:19.521383 (+ 752us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:19.523240 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:19.523243 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:19.523245 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:19.523269 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:19.524526 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:19.524918 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:19.524921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:19.524922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:19.524998 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:19.525279 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:19.525283 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:19.525285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:19.525449 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:19.525566 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:19.525822 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 12:33:19.525939 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260926 12:33:19.527395 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 10746
I20260926 12:33:19.532958 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 10880
I20260926 12:33:19.539608 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 11014
I20260926 12:33:19.544682 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 10652
2026-09-26T12:33:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2158 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:33:19 dist-test-slave-t7tk krb5kdc[11157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:33:19 dist-test-slave-t7tk krb5kdc[11157](info): set up 2 sockets
Sep 26 12:33:19 dist-test-slave-t7tk krb5kdc[11157](info): commencing operation
krb5kdc: starting...
W20260926 12:33:20.514593 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:33:20 dist-test-slave-t7tk krb5kdc[11157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:33:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:33:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:33:20.641602 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:33817
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:37579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:33817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:20.753935 11170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:20.754254 11170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:20.754297 11170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:20.758538 11170 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:20.758615 11170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:20.758634 11170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:20.758651 11170 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:20.758664 11170 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:33:20.764292 11170 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:37579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:33817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:33817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:20.765499 11170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:20.766566 11170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:20.775660 11175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:20.775712 11170 server_base.cc:1034] running on GCE node
W20260926 12:33:20.775744 11176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:20.775728 11178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:20.776526 11170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:20.777729 11170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:20.778909 11170 hybrid_clock.cc:648] HybridClock initialized: now 1790426000778890 us; error 36 us; skew 500 ppm
Sep 26 12:33:20 dist-test-slave-t7tk krb5kdc[11157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426000, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:20.790027 11170 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:33:20.791069 11170 webserver.cc:460] Webserver started at http://127.10.98.62:39309/ using document root <none> and password file <none>
I20260926 12:33:20.791635 11170 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:20.791697 11170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:20.791867 11170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:20.794075 11170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "83a97a98751f435fa32a724ee9b84eb4"
format_stamp: "Formatted at 2026-09-26 12:33:20 on dist-test-slave-t7tk"
server_key: "a88e9167548a438b94d740be8ae5e01e"
server_key_iv: "c3412dc3d284f76711396e62b8350a6b"
server_key_version: "encryptionkey@0"
I20260926 12:33:20.794618 11170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "83a97a98751f435fa32a724ee9b84eb4"
format_stamp: "Formatted at 2026-09-26 12:33:20 on dist-test-slave-t7tk"
server_key: "a88e9167548a438b94d740be8ae5e01e"
server_key_iv: "c3412dc3d284f76711396e62b8350a6b"
server_key_version: "encryptionkey@0"
I20260926 12:33:20.798619 11170 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:33:20.801242 11185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:20.802688 11170 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:33:20.802806 11170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "83a97a98751f435fa32a724ee9b84eb4"
format_stamp: "Formatted at 2026-09-26 12:33:20 on dist-test-slave-t7tk"
server_key: "a88e9167548a438b94d740be8ae5e01e"
server_key_iv: "c3412dc3d284f76711396e62b8350a6b"
server_key_version: "encryptionkey@0"
I20260926 12:33:20.802915 11170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:20.816277 11170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:20.819836 11170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:20.820010 11170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:20.827237 11170 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:33817
I20260926 12:33:20.827266 11237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:33817 every 8 connection(s)
I20260926 12:33:20.828258 11170 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:33:20.831252 11238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:20.837281 11238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4: Bootstrap starting.
I20260926 12:33:20.838408 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 11170
I20260926 12:33:20.838501 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:33:20.838745 10632 external_mini_cluster.cc:1428] Setting key 82a4bb4d7ea069a1befd6a94a0cfca34
I20260926 12:33:20.840126 11238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:20.841017 11238 log.cc:824] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:20.843089 11238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4: No bootstrap required, opened a new log
I20260926 12:33:20.846731 11238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a97a98751f435fa32a724ee9b84eb4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 33817 } }
I20260926 12:33:20.846995 11238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:20.847052 11238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83a97a98751f435fa32a724ee9b84eb4, State: Initialized, Role: FOLLOWER
Sep 26 12:33:20 dist-test-slave-t7tk krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:20.847497 11238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a97a98751f435fa32a724ee9b84eb4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 33817 } }
I20260926 12:33:20.847628 11238 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:33:20.847705 11238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:33:20.847815 11238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:20.848869 11238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a97a98751f435fa32a724ee9b84eb4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 33817 } }
I20260926 12:33:20.849300 11238 leader_election.cc:302] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83a97a98751f435fa32a724ee9b84eb4; no voters:
I20260926 12:33:20.849635 11238 leader_election.cc:288] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:33:20.849746 11243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:20.849956 11243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4 [term 1 LEADER]: Becoming Leader. State: Replica: 83a97a98751f435fa32a724ee9b84eb4, State: Running, Role: LEADER
I20260926 12:33:20.850386 11243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a97a98751f435fa32a724ee9b84eb4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 33817 } }
I20260926 12:33:20.851318 11238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:20.852427 11244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83a97a98751f435fa32a724ee9b84eb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a97a98751f435fa32a724ee9b84eb4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 33817 } } }
I20260926 12:33:20.852573 11244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:20.852850 11245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83a97a98751f435fa32a724ee9b84eb4. Latest consensus state: current_term: 1 leader_uuid: "83a97a98751f435fa32a724ee9b84eb4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83a97a98751f435fa32a724ee9b84eb4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 33817 } } }
I20260926 12:33:20.852937 11245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:20.852998 11252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:33:20.854594 11241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:20.840004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59332 (local address 127.10.98.62:33817)
0926 12:33:20.840496 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:20.840511 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:20.840570 (+ 59us) server_negotiation.cc:407] Connection header received
0926 12:33:20.841331 (+ 761us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:20.841353 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:20.841805 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:20.842180 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:20.843045 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:20.844065 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:20.844910 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:20.845463 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:20.845626 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:20.848191 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:20.848196 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:20.848211 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:20.848243 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:20.850319 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:20.850894 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:20.850901 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:20.850908 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:20.851150 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:20.851534 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:20.851537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:20.851539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:20.852121 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:20.852294 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:20.853218 (+ 924us) server_negotiation.cc:299] Negotiation successful
0926 12:33:20.853553 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":160,"threads_started":1}
I20260926 12:33:20.856421 11252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:33:20.863220 11252 catalog_manager.cc:1347] Generated new cluster ID: c52288b1ca124c2c9d460aefb01fb0d8
I20260926 12:33:20.863317 11252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:33:20.878218 11252 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:33:20.879616 11252 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:33:20.885344 11252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 83a97a98751f435fa32a724ee9b84eb4: Generated new TSK 0
I20260926 12:33:20.886078 11252 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
I20260926 12:33:20.917038 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:33817
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:37579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:21.032390 11264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:21.032680 11264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:21.032743 11264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:21.036964 11264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:21.037052 11264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:21.037159 11264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:33:21.044574 11264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:37579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:33817
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11264
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:21.045804 11264 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:21.046928 11264 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:21.058691 11269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:21.058717 11270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:21.058820 11272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:21.058970 11264 server_base.cc:1034] running on GCE node
I20260926 12:33:21.059478 11264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:21.060150 11264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:21.061339 11264 hybrid_clock.cc:648] HybridClock initialized: now 1790426001061314 us; error 41 us; skew 500 ppm
Sep 26 12:33:21 dist-test-slave-t7tk krb5kdc[11157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426001, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:21.064509 11264 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:33:21.065673 11264 webserver.cc:460] Webserver started at http://127.10.98.1:40599/ using document root <none> and password file <none>
I20260926 12:33:21.066345 11264 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:21.066419 11264 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:21.066618 11264 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:21.069027 11264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "2543f61bdea24a00b7f268de9d387feb"
format_stamp: "Formatted at 2026-09-26 12:33:21 on dist-test-slave-t7tk"
server_key: "7874d0392bc227a671dcd932d065ddfc"
server_key_iv: "f2a63f5566db62f0cd9b6450ca19a15c"
server_key_version: "encryptionkey@0"
I20260926 12:33:21.069640 11264 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "2543f61bdea24a00b7f268de9d387feb"
format_stamp: "Formatted at 2026-09-26 12:33:21 on dist-test-slave-t7tk"
server_key: "7874d0392bc227a671dcd932d065ddfc"
server_key_iv: "f2a63f5566db62f0cd9b6450ca19a15c"
server_key_version: "encryptionkey@0"
I20260926 12:33:21.073899 11264 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:33:21.076608 11279 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:21.078137 11264 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 12:33:21.078249 11264 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "2543f61bdea24a00b7f268de9d387feb"
format_stamp: "Formatted at 2026-09-26 12:33:21 on dist-test-slave-t7tk"
server_key: "7874d0392bc227a671dcd932d065ddfc"
server_key_iv: "f2a63f5566db62f0cd9b6450ca19a15c"
server_key_version: "encryptionkey@0"
I20260926 12:33:21.078344 11264 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:21.098932 11264 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:21.102275 11264 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:21.102454 11264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:21.103052 11264 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:21.103920 11264 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:21.103984 11264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:21.104033 11264 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:21.104054 11264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:21.112739 11264 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:34791
I20260926 12:33:21.112795 11392 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:34791 every 8 connection(s)
I20260926 12:33:21.113847 11264 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:33:21.114040 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 11264
I20260926 12:33:21.114142 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:33:21.114426 10632 external_mini_cluster.cc:1428] Setting key 525efa1301e80d8c5bf6f318fa4ff7d6
WARNING: no policy specified for kudu/127.10.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.2@KRBTEST.COM" created.
Sep 26 12:33:21 dist-test-slave-t7tk krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426001, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:21.132445 11241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:21.115946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:36629 (local address 127.10.98.62:33817)
0926 12:33:21.116068 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:21.116072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:21.117167 (+ 1095us) server_negotiation.cc:407] Connection header received
0926 12:33:21.118582 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:21.118586 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:21.118635 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:21.118725 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:21.120338 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.120893 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.122642 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.122888 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.122992 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:21.128821 (+ 5829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:21.128825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:21.128827 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:21.128859 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:21.130282 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:21.130831 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:21.130835 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:21.130836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:21.130915 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:21.131262 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:21.131265 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:21.131266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:21.131471 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:21.131558 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:21.132108 (+ 550us) server_negotiation.cc:299] Negotiation successful
0926 12:33:21.132275 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 12:33:21.133340 11395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:21.116434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:33817 (local address 127.10.98.1:36629)
0926 12:33:21.116967 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:21.117029 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:21.118345 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:21.118933 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:21.118943 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:21.119423 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:21.120157 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:21.120180 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.121032 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.121035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:21.122494 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:21.122507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.123061 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.123066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:21.123282 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:21.123924 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:21.123948 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:21.128599 (+ 4651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:21.130416 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:21.130424 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:21.130443 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:21.130720 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:21.131003 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:21.131006 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:21.131008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:21.131167 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:21.131572 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:21.131581 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:21.131855 (+ 274us) client_negotiation.cc:769] Sending connection context
0926 12:33:21.132034 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 12:33:21.132402 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":121,"threads_started":1}
I20260926 12:33:21.134676 11393 heartbeater.cc:347] Connected to a master server at 127.10.98.62:33817
I20260926 12:33:21.134907 11393 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:21.135427 11393 heartbeater.cc:511] Master 127.10.98.62:33817 requested a full tablet report, sending...
I20260926 12:33:21.137202 11202 ts_manager.cc:194] Registered new tserver with Master: 2543f61bdea24a00b7f268de9d387feb (127.10.98.1:34791)
I20260926 12:33:21.138392 11202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:36629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
I20260926 12:33:21.144753 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:33817
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:37579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:21.258028 11398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:21.258308 11398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:21.258353 11398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:21.262794 11398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:21.262868 11398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:21.262938 11398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:33:21.268450 11398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:37579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:33817
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11398
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:21.269610 11398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:21.270691 11398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:21.280579 11404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:21.280639 11403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:21.280651 11406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:21.280627 11398 server_base.cc:1034] running on GCE node
I20260926 12:33:21.281340 11398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:21.282102 11398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:21.283257 11398 hybrid_clock.cc:648] HybridClock initialized: now 1790426001283238 us; error 35 us; skew 500 ppm
Sep 26 12:33:21 dist-test-slave-t7tk krb5kdc[11157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426001, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:21.285984 11398 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:33:21.286911 11398 webserver.cc:460] Webserver started at http://127.10.98.2:42757/ using document root <none> and password file <none>
I20260926 12:33:21.287456 11398 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:21.287521 11398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:21.287716 11398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:21.289862 11398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "62ebf13b9c734aed845756867632de8c"
format_stamp: "Formatted at 2026-09-26 12:33:21 on dist-test-slave-t7tk"
server_key: "54c3a8826fe5c3837aa4af80393e67ef"
server_key_iv: "7de0497c180fe68c1cab2092e54c514d"
server_key_version: "encryptionkey@0"
I20260926 12:33:21.290400 11398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "62ebf13b9c734aed845756867632de8c"
format_stamp: "Formatted at 2026-09-26 12:33:21 on dist-test-slave-t7tk"
server_key: "54c3a8826fe5c3837aa4af80393e67ef"
server_key_iv: "7de0497c180fe68c1cab2092e54c514d"
server_key_version: "encryptionkey@0"
I20260926 12:33:21.294354 11398 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260926 12:33:21.297000 11413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:21.298564 11398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:33:21.298709 11398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "62ebf13b9c734aed845756867632de8c"
format_stamp: "Formatted at 2026-09-26 12:33:21 on dist-test-slave-t7tk"
server_key: "54c3a8826fe5c3837aa4af80393e67ef"
server_key_iv: "7de0497c180fe68c1cab2092e54c514d"
server_key_version: "encryptionkey@0"
I20260926 12:33:21.298877 11398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:21.320658 11398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:21.324107 11398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:21.324290 11398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:21.325017 11398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:21.326006 11398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:21.326050 11398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:21.326097 11398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:21.326115 11398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:21.336642 11398 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:39449
I20260926 12:33:21.336656 11526 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:39449 every 8 connection(s)
I20260926 12:33:21.337806 11398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:33:21.343286 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 11398
I20260926 12:33:21.343402 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:33:21.343683 10632 external_mini_cluster.cc:1428] Setting key 7ee982a845cfe9a9508e85aa13144dc5
Sep 26 12:33:21 dist-test-slave-t7tk krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426001, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:21.351933 11241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:21.339768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:34035 (local address 127.10.98.62:33817)
0926 12:33:21.339918 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:21.339922 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:21.340835 (+ 913us) server_negotiation.cc:407] Connection header received
0926 12:33:21.342140 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:21.342143 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:21.342191 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:21.342259 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:21.343906 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.344543 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.345814 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.346031 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.346120 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:21.348513 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:21.348516 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:21.348518 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:21.348543 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:21.349903 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:21.350400 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:21.350403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:21.350405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:21.350481 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:21.350792 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:21.350796 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:21.350797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:21.350976 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:21.351062 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:21.351635 (+ 573us) server_negotiation.cc:299] Negotiation successful
0926 12:33:21.351769 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:33:21.353048 11529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:21.340156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:33817 (local address 127.10.98.2:34035)
0926 12:33:21.340654 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:21.340726 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:21.341860 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:21.342486 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:21.342496 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:21.342992 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:21.343690 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:21.343711 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.344678 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.344681 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:21.345699 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:21.345710 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.346173 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.346177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:21.346345 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:21.346898 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:21.346915 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:21.348345 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:21.350015 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:21.350021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:21.350039 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:21.350305 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:21.350561 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:21.350564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:21.350566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:21.350709 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:21.351073 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:21.351080 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:21.351340 (+ 260us) client_negotiation.cc:769] Sending connection context
0926 12:33:21.351589 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 12:33:21.351925 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":112,"threads_started":1}
I20260926 12:33:21.354225 11527 heartbeater.cc:347] Connected to a master server at 127.10.98.62:33817
I20260926 12:33:21.354463 11527 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.10.98.3@KRBTEST.COM; defaulting to no policy
I20260926 12:33:21.354964 11527 heartbeater.cc:511] Master 127.10.98.62:33817 requested a full tablet report, sending...
I20260926 12:33:21.355974 11202 ts_manager.cc:194] Registered new tserver with Master: 62ebf13b9c734aed845756867632de8c (127.10.98.2:39449)
I20260926 12:33:21.356527 11202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:34035
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
I20260926 12:33:21.376976 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:33817
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:37579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:21.491974 11532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:21.492259 11532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:21.492321 11532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:21.496598 11532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:21.496675 11532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:21.496765 11532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:33:21.502270 11532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:37579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:33817
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11532
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:21.503314 11532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:21.504346 11532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:21.513793 11537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:21.514042 11540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:21.514019 11538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:21.514324 11532 server_base.cc:1034] running on GCE node
I20260926 12:33:21.514755 11532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:21.515420 11532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:21.516587 11532 hybrid_clock.cc:648] HybridClock initialized: now 1790426001516570 us; error 31 us; skew 500 ppm
Sep 26 12:33:21 dist-test-slave-t7tk krb5kdc[11157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426001, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:21.519481 11532 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:33:21.520604 11532 webserver.cc:460] Webserver started at http://127.10.98.3:45661/ using document root <none> and password file <none>
I20260926 12:33:21.521220 11532 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:21.521296 11532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:21.521497 11532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:21.523550 11532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "7914b30ab872451ea130943bc2cb995a"
format_stamp: "Formatted at 2026-09-26 12:33:21 on dist-test-slave-t7tk"
server_key: "01dfd49b6400c2e88439a5aca710cc3a"
server_key_iv: "d2e0d502b1aaad4516b3fa987f649c82"
server_key_version: "encryptionkey@0"
I20260926 12:33:21.524099 11532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "7914b30ab872451ea130943bc2cb995a"
format_stamp: "Formatted at 2026-09-26 12:33:21 on dist-test-slave-t7tk"
server_key: "01dfd49b6400c2e88439a5aca710cc3a"
server_key_iv: "d2e0d502b1aaad4516b3fa987f649c82"
server_key_version: "encryptionkey@0"
I20260926 12:33:21.527915 11532 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:33:21.530462 11547 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:21.531750 11532 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:33:21.531867 11532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "7914b30ab872451ea130943bc2cb995a"
format_stamp: "Formatted at 2026-09-26 12:33:21 on dist-test-slave-t7tk"
server_key: "01dfd49b6400c2e88439a5aca710cc3a"
server_key_iv: "d2e0d502b1aaad4516b3fa987f649c82"
server_key_version: "encryptionkey@0"
I20260926 12:33:21.531963 11532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:21.559206 11532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:21.562824 11532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:21.563016 11532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:21.563644 11532 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:21.564509 11532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:21.564565 11532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:21.564620 11532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:21.564649 11532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:21.573421 11532 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:36173
I20260926 12:33:21.573546 11660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:36173 every 8 connection(s)
I20260926 12:33:21.574364 11532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
Sep 26 12:33:21 dist-test-slave-t7tk krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426001, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:21.583910 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 11532
I20260926 12:33:21.584030 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:33:21.584340 10632 external_mini_cluster.cc:1428] Setting key 2bf5feb14e2ae8c2ae138f868d3ae610
I20260926 12:33:21.587836 11241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:21.576317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:54067 (local address 127.10.98.62:33817)
0926 12:33:21.576467 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:21.576471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:21.577433 (+ 962us) server_negotiation.cc:407] Connection header received
0926 12:33:21.578315 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:21.578318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:21.578373 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:21.578445 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:21.579609 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.580124 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.581286 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.581504 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.581599 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:21.584191 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:21.584194 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:21.584196 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:21.584223 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:21.585743 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:21.586326 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:21.586330 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:21.586332 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:21.586456 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:21.586824 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:21.586827 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:21.586835 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:21.587024 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:21.587109 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:21.587613 (+ 504us) server_negotiation.cc:299] Negotiation successful
0926 12:33:21.587718 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 12:33:21.588632 11663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:21.576765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:33817 (local address 127.10.98.3:54067)
0926 12:33:21.577296 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:21.577341 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:21.578121 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:21.578612 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:21.578619 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:21.578961 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:21.579465 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:21.579478 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.580239 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.580242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:21.581180 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:21.581189 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.581639 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.581643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:21.581807 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:21.582407 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:21.582422 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:21.584017 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:21.585846 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:21.585852 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:21.585868 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:21.586203 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:21.586553 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:21.586556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:21.586558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:21.586712 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:21.587124 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:21.587129 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:21.587365 (+ 236us) client_negotiation.cc:769] Sending connection context
0926 12:33:21.587526 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 12:33:21.587811 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":122,"threads_started":1}
I20260926 12:33:21.589808 11661 heartbeater.cc:347] Connected to a master server at 127.10.98.62:33817
I20260926 12:33:21.590116 11661 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:21.590634 11661 heartbeater.cc:511] Master 127.10.98.62:33817 requested a full tablet report, sending...
I20260926 12:33:21.591683 11202 ts_manager.cc:194] Registered new tserver with Master: 7914b30ab872451ea130943bc2cb995a (127.10.98.3:36173)
I20260926 12:33:21.592270 11202 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:54067
I20260926 12:33:21.598397 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:33:21.608934 11241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:21.601969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59342 (local address 127.10.98.62:33817)
0926 12:33:21.602083 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:21.602087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:21.602221 (+ 134us) server_negotiation.cc:407] Connection header received
0926 12:33:21.602271 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:21.602273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:21.602317 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:21.602390 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:21.603252 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.603792 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.604691 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.604894 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.604965 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:21.606365 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:21.606368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:21.606370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:21.606398 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:21.607583 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:21.607920 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:21.607923 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:21.607924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:21.608002 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:21.608245 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:21.608247 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:21.608248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:21.608459 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:21.608560 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:21.608716 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 12:33:21.608807 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:33:21.613947 11202 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59342:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:33:21.616935 11202 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:33:21.631006 11674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:21.625698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:39449 (local address 127.0.0.1:54036)
0926 12:33:21.625953 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:21.625966 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:21.626152 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:21.626839 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:21.626842 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:21.626869 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:21.626968 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:21.626974 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.628497 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.628500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:21.629405 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:21.629415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.630466 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.630469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:21.630579 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:21.630619 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 12:33:21.630720 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 12:33:21.630844 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":54,"thread_start_us":76,"threads_started":1}
I20260926 12:33:21.631210 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:21.626490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41414 (local address 127.10.98.1:34791)
0926 12:33:21.626871 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:21.626887 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:21.626908 (+ 21us) server_negotiation.cc:407] Connection header received
0926 12:33:21.626976 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:21.626980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:21.627150 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:21.627281 (+ 131us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:21.627599 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.628373 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.629843 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.630637 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.630739 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:21.630883 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:21.630946 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 12:33:21.631065 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":113,"threads_started":1}
I20260926 12:33:21.631304 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:21.625916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54036 (local address 127.10.98.2:39449)
0926 12:33:21.626400 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:21.626413 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:21.626436 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:33:21.626508 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:21.626513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:21.626701 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:21.627129 (+ 428us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:21.627306 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.628353 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.629542 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.630339 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.630438 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:21.630592 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:21.631075 (+ 483us) server_negotiation.cc:299] Negotiation successful
0926 12:33:21.631179 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":118,"threads_started":1}
I20260926 12:33:21.631331 11673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:21.625150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34424 (local address 127.10.98.3:36173)
0926 12:33:21.625595 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:21.625608 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:21.625633 (+ 25us) server_negotiation.cc:407] Connection header received
0926 12:33:21.625710 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:21.625716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:21.625962 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:21.626125 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:21.626617 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.627686 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.629558 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.630367 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.630470 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:21.630625 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:21.631101 (+ 476us) server_negotiation.cc:299] Negotiation successful
0926 12:33:21.631193 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":138,"threads_started":1}
I20260926 12:33:21.631733 11676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:21.626075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:34791 (local address 127.0.0.1:41414)
0926 12:33:21.626341 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:21.626353 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:21.626431 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:21.627337 (+ 906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:21.627340 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:21.627358 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:21.627440 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:21.627446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.628527 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.628530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:21.629717 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:21.629726 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.630750 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.630754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:21.630840 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:21.630861 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:33:21.630917 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 12:33:21.630987 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":32,"thread_start_us":85,"threads_started":1}
I20260926 12:33:21.631740 11672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:21.624901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:36173 (local address 127.0.0.1:34424)
0926 12:33:21.625220 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:21.625260 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:21.625401 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:21.626324 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:21.626330 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:21.626353 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:21.626485 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:21.626491 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.627829 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.627832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:21.629401 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:21.629412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.630693 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.630697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:21.630797 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:21.630820 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:33:21.630909 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 12:33:21.630986 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":67,"thread_start_us":73,"threads_started":1}
I20260926 12:33:21.634721 11460 tablet_service.cc:1467] Processing CreateTablet for tablet 3803aae6636245acbaf9b846a0a592ce (DEFAULT_TABLE table=test-table [id=c7ba3d18c03545f0a29d0c01ea2a4f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:21.634871 11595 tablet_service.cc:1467] Processing CreateTablet for tablet 3803aae6636245acbaf9b846a0a592ce (DEFAULT_TABLE table=test-table [id=c7ba3d18c03545f0a29d0c01ea2a4f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:21.634847 11327 tablet_service.cc:1467] Processing CreateTablet for tablet 3803aae6636245acbaf9b846a0a592ce (DEFAULT_TABLE table=test-table [id=c7ba3d18c03545f0a29d0c01ea2a4f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:21.635812 11460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3803aae6636245acbaf9b846a0a592ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:21.635903 11327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3803aae6636245acbaf9b846a0a592ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:21.635903 11595 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3803aae6636245acbaf9b846a0a592ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:21.642105 11678 tablet_bootstrap.cc:492] T 3803aae6636245acbaf9b846a0a592ce P 62ebf13b9c734aed845756867632de8c: Bootstrap starting.
I20260926 12:33:21.643926 11680 tablet_bootstrap.cc:492] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb: Bootstrap starting.
I20260926 12:33:21.644879 11678 tablet_bootstrap.cc:654] T 3803aae6636245acbaf9b846a0a592ce P 62ebf13b9c734aed845756867632de8c: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:21.645447 11679 tablet_bootstrap.cc:492] T 3803aae6636245acbaf9b846a0a592ce P 7914b30ab872451ea130943bc2cb995a: Bootstrap starting.
I20260926 12:33:21.645802 11678 log.cc:824] T 3803aae6636245acbaf9b846a0a592ce P 62ebf13b9c734aed845756867632de8c: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:21.646224 11680 tablet_bootstrap.cc:654] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:21.647153 11680 log.cc:824] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:21.647841 11678 tablet_bootstrap.cc:492] T 3803aae6636245acbaf9b846a0a592ce P 62ebf13b9c734aed845756867632de8c: No bootstrap required, opened a new log
I20260926 12:33:21.648067 11678 ts_tablet_manager.cc:1397] T 3803aae6636245acbaf9b846a0a592ce P 62ebf13b9c734aed845756867632de8c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 12:33:21.648437 11679 tablet_bootstrap.cc:654] T 3803aae6636245acbaf9b846a0a592ce P 7914b30ab872451ea130943bc2cb995a: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:21.649343 11679 log.cc:824] T 3803aae6636245acbaf9b846a0a592ce P 7914b30ab872451ea130943bc2cb995a: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:21.649467 11680 tablet_bootstrap.cc:492] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb: No bootstrap required, opened a new log
I20260926 12:33:21.649649 11680 ts_tablet_manager.cc:1397] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:33:21.651439 11678 raft_consensus.cc:348] T 3803aae6636245acbaf9b846a0a592ce P 62ebf13b9c734aed845756867632de8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7914b30ab872451ea130943bc2cb995a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36173 } } peers { permanent_uuid: "62ebf13b9c734aed845756867632de8c" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 39449 } } peers { permanent_uuid: "2543f61bdea24a00b7f268de9d387feb" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34791 } }
I20260926 12:33:21.651695 11678 raft_consensus.cc:374] T 3803aae6636245acbaf9b846a0a592ce P 62ebf13b9c734aed845756867632de8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:21.651755 11678 raft_consensus.cc:729] T 3803aae6636245acbaf9b846a0a592ce P 62ebf13b9c734aed845756867632de8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ebf13b9c734aed845756867632de8c, State: Initialized, Role: FOLLOWER
I20260926 12:33:21.652242 11678 consensus_queue.cc:260] T 3803aae6636245acbaf9b846a0a592ce P 62ebf13b9c734aed845756867632de8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7914b30ab872451ea130943bc2cb995a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36173 } } peers { permanent_uuid: "62ebf13b9c734aed845756867632de8c" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 39449 } } peers { permanent_uuid: "2543f61bdea24a00b7f268de9d387feb" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34791 } }
I20260926 12:33:21.652968 11527 heartbeater.cc:503] Master 127.10.98.62:33817 was elected leader, sending a full tablet report...
I20260926 12:33:21.653297 11678 ts_tablet_manager.cc:1428] T 3803aae6636245acbaf9b846a0a592ce P 62ebf13b9c734aed845756867632de8c: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260926 12:33:21.653560 11680 raft_consensus.cc:348] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7914b30ab872451ea130943bc2cb995a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36173 } } peers { permanent_uuid: "62ebf13b9c734aed845756867632de8c" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 39449 } } peers { permanent_uuid: "2543f61bdea24a00b7f268de9d387feb" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34791 } }
I20260926 12:33:21.653836 11680 raft_consensus.cc:374] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:21.653892 11680 raft_consensus.cc:729] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2543f61bdea24a00b7f268de9d387feb, State: Initialized, Role: FOLLOWER
I20260926 12:33:21.654313 11680 consensus_queue.cc:260] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7914b30ab872451ea130943bc2cb995a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36173 } } peers { permanent_uuid: "62ebf13b9c734aed845756867632de8c" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 39449 } } peers { permanent_uuid: "2543f61bdea24a00b7f268de9d387feb" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34791 } }
I20260926 12:33:21.654989 11393 heartbeater.cc:503] Master 127.10.98.62:33817 was elected leader, sending a full tablet report...
I20260926 12:33:21.655359 11680 ts_tablet_manager.cc:1428] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:33:21.656364 11679 tablet_bootstrap.cc:492] T 3803aae6636245acbaf9b846a0a592ce P 7914b30ab872451ea130943bc2cb995a: No bootstrap required, opened a new log
I20260926 12:33:21.656580 11679 ts_tablet_manager.cc:1397] T 3803aae6636245acbaf9b846a0a592ce P 7914b30ab872451ea130943bc2cb995a: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260926 12:33:21.659788 11679 raft_consensus.cc:348] T 3803aae6636245acbaf9b846a0a592ce P 7914b30ab872451ea130943bc2cb995a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7914b30ab872451ea130943bc2cb995a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36173 } } peers { permanent_uuid: "62ebf13b9c734aed845756867632de8c" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 39449 } } peers { permanent_uuid: "2543f61bdea24a00b7f268de9d387feb" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34791 } }
I20260926 12:33:21.660045 11679 raft_consensus.cc:374] T 3803aae6636245acbaf9b846a0a592ce P 7914b30ab872451ea130943bc2cb995a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:21.660113 11679 raft_consensus.cc:729] T 3803aae6636245acbaf9b846a0a592ce P 7914b30ab872451ea130943bc2cb995a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7914b30ab872451ea130943bc2cb995a, State: Initialized, Role: FOLLOWER
I20260926 12:33:21.660514 11679 consensus_queue.cc:260] T 3803aae6636245acbaf9b846a0a592ce P 7914b30ab872451ea130943bc2cb995a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7914b30ab872451ea130943bc2cb995a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36173 } } peers { permanent_uuid: "62ebf13b9c734aed845756867632de8c" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 39449 } } peers { permanent_uuid: "2543f61bdea24a00b7f268de9d387feb" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34791 } }
I20260926 12:33:21.661288 11661 heartbeater.cc:503] Master 127.10.98.62:33817 was elected leader, sending a full tablet report...
I20260926 12:33:21.661537 11679 ts_tablet_manager.cc:1428] T 3803aae6636245acbaf9b846a0a592ce P 7914b30ab872451ea130943bc2cb995a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:33:21.768182 11685 raft_consensus.cc:482] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:33:21.768404 11685 raft_consensus.cc:504] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7914b30ab872451ea130943bc2cb995a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36173 } } peers { permanent_uuid: "62ebf13b9c734aed845756867632de8c" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 39449 } } peers { permanent_uuid: "2543f61bdea24a00b7f268de9d387feb" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34791 } }
I20260926 12:33:21.769662 11685 leader_election.cc:288] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7914b30ab872451ea130943bc2cb995a (127.10.98.3:36173), 62ebf13b9c734aed845756867632de8c (127.10.98.2:39449)
I20260926 12:33:21.773864 11688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:21.770409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:39449 (local address 127.10.98.1:47861)
0926 12:33:21.770696 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:21.770709 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:21.770810 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:21.771029 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:21.771032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:21.771045 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:21.771177 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:21.771183 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.771858 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.771860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:21.772748 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:21.772757 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.773449 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.773453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:21.773547 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:21.773570 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:33:21.773639 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 12:33:21.773696 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":21,"thread_start_us":101,"threads_started":1}
I20260926 12:33:21.773865 11675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:21.770523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:47861 (local address 127.10.98.2:39449)
0926 12:33:21.770667 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:21.770672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:21.770834 (+ 162us) server_negotiation.cc:407] Connection header received
0926 12:33:21.770887 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:21.770890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:21.770934 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:21.771007 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:21.771260 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.771760 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.772870 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.773341 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.773419 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:21.773486 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:21.773663 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 12:33:21.773744 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 12:33:21.773922 11687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:21.769929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:36173 (local address 127.10.98.1:49247)
0926 12:33:21.770231 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:21.770246 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:21.770373 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:21.770852 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:21.770855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:21.770877 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:21.771014 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:21.771021 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.771797 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.771801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:21.772750 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:21.772759 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.773532 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.773536 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:21.773629 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:21.773652 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:33:21.773724 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 12:33:21.773785 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":47,"thread_start_us":117,"threads_started":1}
I20260926 12:33:21.774467 11673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:21.770040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:49247 (local address 127.10.98.3:36173)
0926 12:33:21.770230 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:21.770233 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:21.770324 (+ 91us) server_negotiation.cc:407] Connection header received
0926 12:33:21.770678 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:21.770681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:21.770739 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:21.770814 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:21.771128 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.771687 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.772889 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.773413 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.773512 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:21.773582 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:21.773752 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 12:33:21.773814 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260926 12:33:21.774863 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3803aae6636245acbaf9b846a0a592ce" candidate_uuid: "2543f61bdea24a00b7f268de9d387feb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62ebf13b9c734aed845756867632de8c" is_pre_election: true
I20260926 12:33:21.775040 11615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3803aae6636245acbaf9b846a0a592ce" candidate_uuid: "2543f61bdea24a00b7f268de9d387feb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7914b30ab872451ea130943bc2cb995a" is_pre_election: true
I20260926 12:33:21.775182 11474 raft_consensus.cc:2454] T 3803aae6636245acbaf9b846a0a592ce P 62ebf13b9c734aed845756867632de8c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2543f61bdea24a00b7f268de9d387feb in term 0.
I20260926 12:33:21.775331 11615 raft_consensus.cc:2454] T 3803aae6636245acbaf9b846a0a592ce P 7914b30ab872451ea130943bc2cb995a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2543f61bdea24a00b7f268de9d387feb in term 0.
I20260926 12:33:21.775707 11280 leader_election.cc:302] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2543f61bdea24a00b7f268de9d387feb, 62ebf13b9c734aed845756867632de8c; no voters:
I20260926 12:33:21.775954 11685 raft_consensus.cc:2781] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:33:21.776047 11685 raft_consensus.cc:482] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:33:21.776091 11685 raft_consensus.cc:3037] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:21.777128 11685 raft_consensus.cc:504] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7914b30ab872451ea130943bc2cb995a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36173 } } peers { permanent_uuid: "62ebf13b9c734aed845756867632de8c" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 39449 } } peers { permanent_uuid: "2543f61bdea24a00b7f268de9d387feb" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34791 } }
I20260926 12:33:21.777438 11685 leader_election.cc:288] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb [CANDIDATE]: Term 1 election: Requested vote from peers 7914b30ab872451ea130943bc2cb995a (127.10.98.3:36173), 62ebf13b9c734aed845756867632de8c (127.10.98.2:39449)
I20260926 12:33:21.777796 11615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3803aae6636245acbaf9b846a0a592ce" candidate_uuid: "2543f61bdea24a00b7f268de9d387feb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7914b30ab872451ea130943bc2cb995a"
I20260926 12:33:21.777870 11474 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3803aae6636245acbaf9b846a0a592ce" candidate_uuid: "2543f61bdea24a00b7f268de9d387feb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62ebf13b9c734aed845756867632de8c"
I20260926 12:33:21.777909 11615 raft_consensus.cc:3037] T 3803aae6636245acbaf9b846a0a592ce P 7914b30ab872451ea130943bc2cb995a [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:21.778002 11474 raft_consensus.cc:3037] T 3803aae6636245acbaf9b846a0a592ce P 62ebf13b9c734aed845756867632de8c [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:21.778983 11615 raft_consensus.cc:2454] T 3803aae6636245acbaf9b846a0a592ce P 7914b30ab872451ea130943bc2cb995a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2543f61bdea24a00b7f268de9d387feb in term 1.
I20260926 12:33:21.779063 11474 raft_consensus.cc:2454] T 3803aae6636245acbaf9b846a0a592ce P 62ebf13b9c734aed845756867632de8c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2543f61bdea24a00b7f268de9d387feb in term 1.
I20260926 12:33:21.779317 11280 leader_election.cc:302] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2543f61bdea24a00b7f268de9d387feb, 7914b30ab872451ea130943bc2cb995a; no voters:
I20260926 12:33:21.779551 11685 raft_consensus.cc:2781] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:21.779832 11685 raft_consensus.cc:686] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb [term 1 LEADER]: Becoming Leader. State: Replica: 2543f61bdea24a00b7f268de9d387feb, State: Running, Role: LEADER
I20260926 12:33:21.780174 11685 consensus_queue.cc:237] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7914b30ab872451ea130943bc2cb995a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36173 } } peers { permanent_uuid: "62ebf13b9c734aed845756867632de8c" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 39449 } } peers { permanent_uuid: "2543f61bdea24a00b7f268de9d387feb" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34791 } }
I20260926 12:33:21.783694 11202 catalog_manager.cc:5495] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb reported cstate change: term changed from 0 to 1, leader changed from <none> to 2543f61bdea24a00b7f268de9d387feb (127.10.98.1). New cstate: current_term: 1 leader_uuid: "2543f61bdea24a00b7f268de9d387feb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7914b30ab872451ea130943bc2cb995a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62ebf13b9c734aed845756867632de8c" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 39449 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2543f61bdea24a00b7f268de9d387feb" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34791 } health_report { overall_health: HEALTHY } } }
W20260926 12:33:21.825301 11662 tablet.cc:2369] T 3803aae6636245acbaf9b846a0a592ce P 7914b30ab872451ea130943bc2cb995a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 12:33:21 dist-test-slave-t7tk krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.1@KRBTEST.COM
I20260926 12:33:21.833642 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:21.824013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41416 (local address 127.10.98.1:34791)
0926 12:33:21.824164 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:21.824170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:21.824305 (+ 135us) server_negotiation.cc:407] Connection header received
0926 12:33:21.824433 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:21.824436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:21.824482 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:21.824548 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:21.825243 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.825899 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.826712 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.826989 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.827131 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:21.829569 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:21.829586 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:21.829602 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:21.829640 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:21.831675 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:21.832090 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:21.832096 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:21.832098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:21.832196 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:21.832497 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:21.832500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:21.832502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:21.832809 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:21.833009 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:21.833311 (+ 302us) server_negotiation.cc:299] Negotiation successful
0926 12:33:21.833434 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 12:33:21.838970 11528 tablet.cc:2369] T 3803aae6636245acbaf9b846a0a592ce P 62ebf13b9c734aed845756867632de8c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 12:33:21 dist-test-slave-t7tk krb5kdc[11157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426001, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 12:33:21 dist-test-slave-t7tk krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426001, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.98.1@KRBTEST.COM
I20260926 12:33:21.860349 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:21.851550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41432 (local address 127.10.98.1:34791)
0926 12:33:21.851740 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:21.851745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:21.851830 (+ 85us) server_negotiation.cc:407] Connection header received
0926 12:33:21.851975 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:21.851978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:21.852091 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:21.852166 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:21.852956 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.853614 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.854539 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.854767 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.854855 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:21.857294 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:21.857300 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:21.857303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:21.857330 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:21.858537 (+ 1207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:21.859003 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:21.859007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:21.859008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:21.859090 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:21.859369 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:21.859372 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:21.859373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:21.859564 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:21.859692 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:21.860009 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 12:33:21.860225 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20260926 12:33:21.865321 11394 tablet.cc:2369] T 3803aae6636245acbaf9b846a0a592ce P 2543f61bdea24a00b7f268de9d387feb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 12:33:21 dist-test-slave-t7tk krb5kdc[11157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 12:33:21 dist-test-slave-t7tk krb5kdc[11157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.1@KRBTEST.COM
I20260926 12:33:21.885371 11677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:21.876955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.10.98.1:34791)
0926 12:33:21.877145 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:21.877151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:21.877296 (+ 145us) server_negotiation.cc:407] Connection header received
0926 12:33:21.877445 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:21.877448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:21.877491 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:21.877558 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:21.878248 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.878743 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.879553 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:21.879777 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:21.879870 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:21.882317 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:21.882320 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:21.882322 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:21.882347 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:21.883699 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:21.884134 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:21.884137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:21.884138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:21.884207 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:21.884507 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:21.884509 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:21.884510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:21.884694 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:21.884805 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:21.885054 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 12:33:21.885211 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260926 12:33:21.886785 11307 tablet_service.cc:2908] Rejecting scan request for tablet 3803aae6636245acbaf9b846a0a592ce: Uninitialized: clean time has not yet been initialized
I20260926 12:33:21.888669 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 11264
I20260926 12:33:21.894742 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 11398
I20260926 12:33:21.899431 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 11532
I20260926 12:33:21.904151 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 11170
2026-09-26T12:33:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2359 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:33:21 dist-test-slave-t7tk krb5kdc[11703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:33:21 dist-test-slave-t7tk krb5kdc[11703](info): set up 2 sockets
Sep 26 12:33:21 dist-test-slave-t7tk krb5kdc[11703](info): commencing operation
krb5kdc: starting...
W20260926 12:33:22.866727 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:33:22 dist-test-slave-t7tk krb5kdc[11703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:33:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:33:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:33:22.985267 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:43177
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:34981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:43177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:23.098202 11716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:23.098524 11716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:23.098582 11716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:23.102766 11716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:23.102840 11716 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 12:33:23.102864 11716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:23.102887 11716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:23.102908 11716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:23.102947 11716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:33:23.108433 11716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:34981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:43177
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:43177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:23.109622 11716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:23.110677 11716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:23.119784 11724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:23.119778 11721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:23.119778 11722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:23.119830 11716 server_base.cc:1034] running on GCE node
I20260926 12:33:23.120836 11716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:23.122144 11716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:23.123355 11716 hybrid_clock.cc:648] HybridClock initialized: now 1790426003123337 us; error 36 us; skew 500 ppm
Sep 26 12:33:23 dist-test-slave-t7tk krb5kdc[11703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426003, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:23.126632 11716 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:33:23.127724 11716 webserver.cc:460] Webserver started at http://127.10.98.62:40011/ using document root <none> and password file <none>
I20260926 12:33:23.128299 11716 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:23.128362 11716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:23.128563 11716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:23.130829 11716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "c7cd35a462414c5abe243c2c5de50ad4"
format_stamp: "Formatted at 2026-09-26 12:33:23 on dist-test-slave-t7tk"
server_key: "24057888ad9da3d504c8d45422830b8b"
server_key_iv: "618230f707d8be7da1c9ef8e33b9715b"
server_key_version: "encryptionkey@0"
I20260926 12:33:23.131496 11716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "c7cd35a462414c5abe243c2c5de50ad4"
format_stamp: "Formatted at 2026-09-26 12:33:23 on dist-test-slave-t7tk"
server_key: "24057888ad9da3d504c8d45422830b8b"
server_key_iv: "618230f707d8be7da1c9ef8e33b9715b"
server_key_version: "encryptionkey@0"
I20260926 12:33:23.135890 11716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:33:23.138954 11731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:23.140311 11716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:33:23.140439 11716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "c7cd35a462414c5abe243c2c5de50ad4"
format_stamp: "Formatted at 2026-09-26 12:33:23 on dist-test-slave-t7tk"
server_key: "24057888ad9da3d504c8d45422830b8b"
server_key_iv: "618230f707d8be7da1c9ef8e33b9715b"
server_key_version: "encryptionkey@0"
I20260926 12:33:23.140568 11716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:23.154249 11716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:23.158447 11716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:23.158653 11716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:23.167461 11716 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:43177
I20260926 12:33:23.167520 11793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:43177 every 8 connection(s)
I20260926 12:33:23.168524 11716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:33:23.171572 11794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:23.171768 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 11716
I20260926 12:33:23.171878 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:33:23.172180 10632 external_mini_cluster.cc:1428] Setting key 0e2f52a287b789ff2ee2fe7e08a921a1
I20260926 12:33:23.179082 11794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4: Bootstrap starting.
Sep 26 12:33:23 dist-test-slave-t7tk krb5kdc[11703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:23.181876 11794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:23.182793 11794 log.cc:824] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:23.184875 11794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4: No bootstrap required, opened a new log
I20260926 12:33:23.187815 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:23.173655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35458 (local address 127.10.98.62:43177)
0926 12:33:23.174116 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:23.174137 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:23.174182 (+ 45us) server_negotiation.cc:407] Connection header received
0926 12:33:23.174968 (+ 786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:23.174996 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:23.175471 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:23.175887 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:23.176574 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.178083 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.179015 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.179681 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.179842 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:23.182371 (+ 2529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:23.182378 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:23.182396 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:23.182431 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:23.184761 (+ 2330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:23.185335 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:23.185341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:23.185346 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:23.185473 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:23.185757 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:23.185759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:23.185761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:23.186186 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:23.186370 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:23.186687 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 12:33:23.186983 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":130,"threads_started":1}
I20260926 12:33:23.188427 11794 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7cd35a462414c5abe243c2c5de50ad4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43177 } }
I20260926 12:33:23.188699 11794 raft_consensus.cc:374] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:23.188786 11794 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7cd35a462414c5abe243c2c5de50ad4, State: Initialized, Role: FOLLOWER
I20260926 12:33:23.189321 11794 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7cd35a462414c5abe243c2c5de50ad4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43177 } }
I20260926 12:33:23.189566 11794 raft_consensus.cc:388] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:33:23.189648 11794 raft_consensus.cc:482] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:33:23.189747 11794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:23.190857 11794 raft_consensus.cc:504] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7cd35a462414c5abe243c2c5de50ad4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43177 } }
I20260926 12:33:23.191246 11794 leader_election.cc:302] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c7cd35a462414c5abe243c2c5de50ad4; no voters:
I20260926 12:33:23.191613 11794 leader_election.cc:288] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:33:23.191761 11799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:23.192085 11799 raft_consensus.cc:686] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4 [term 1 LEADER]: Becoming Leader. State: Replica: c7cd35a462414c5abe243c2c5de50ad4, State: Running, Role: LEADER
I20260926 12:33:23.192564 11799 consensus_queue.cc:237] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7cd35a462414c5abe243c2c5de50ad4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43177 } }
I20260926 12:33:23.193226 11794 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:23.194015 11800 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c7cd35a462414c5abe243c2c5de50ad4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7cd35a462414c5abe243c2c5de50ad4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43177 } } }
I20260926 12:33:23.194168 11800 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:23.194015 11799 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c7cd35a462414c5abe243c2c5de50ad4. Latest consensus state: current_term: 1 leader_uuid: "c7cd35a462414c5abe243c2c5de50ad4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c7cd35a462414c5abe243c2c5de50ad4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43177 } } }
I20260926 12:33:23.194669 11799 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:23.195182 11808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:33:23.198731 11808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:33:23.204947 11808 catalog_manager.cc:1347] Generated new cluster ID: 4c3a2f47db3b45f8b59dcf2d34277b78
I20260926 12:33:23.205052 11808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:33:23.231848 11808 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:33:23.233304 11808 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:33:23.240608 11808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4: Generated new TSK 0
I20260926 12:33:23.241632 11808 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
I20260926 12:33:23.280757 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:43177
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:34981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:23.400202 11820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:23.400506 11820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:23.400579 11820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:23.404834 11820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:23.404929 11820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:23.404989 11820 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:33:23.405050 11820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:33:23.410773 11820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:34981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.98.62:43177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:23.411893 11820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:23.412923 11820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:23.422780 11826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:23.422853 11825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:23.423048 11820 server_base.cc:1034] running on GCE node
W20260926 12:33:23.422787 11828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:23.423601 11820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:23.424254 11820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:23.425420 11820 hybrid_clock.cc:648] HybridClock initialized: now 1790426003425388 us; error 51 us; skew 500 ppm
Sep 26 12:33:23 dist-test-slave-t7tk krb5kdc[11703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426003, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:23.428237 11820 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:33:23.429168 11820 webserver.cc:460] Webserver started at http://127.10.98.1:39345/ using document root <none> and password file <none>
I20260926 12:33:23.429739 11820 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:23.429803 11820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:23.429994 11820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:23.432111 11820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "a774498f426c4efdadba4d25a124c4ae"
format_stamp: "Formatted at 2026-09-26 12:33:23 on dist-test-slave-t7tk"
server_key: "875ba2dc2f476f6e19a11972d6a30a55"
server_key_iv: "a84ad484f227abe36a6e94a517dd8c0a"
server_key_version: "encryptionkey@0"
I20260926 12:33:23.432679 11820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "a774498f426c4efdadba4d25a124c4ae"
format_stamp: "Formatted at 2026-09-26 12:33:23 on dist-test-slave-t7tk"
server_key: "875ba2dc2f476f6e19a11972d6a30a55"
server_key_iv: "a84ad484f227abe36a6e94a517dd8c0a"
server_key_version: "encryptionkey@0"
I20260926 12:33:23.436556 11820 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:33:23.439142 11835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:23.440497 11820 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:33:23.440603 11820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "a774498f426c4efdadba4d25a124c4ae"
format_stamp: "Formatted at 2026-09-26 12:33:23 on dist-test-slave-t7tk"
server_key: "875ba2dc2f476f6e19a11972d6a30a55"
server_key_iv: "a84ad484f227abe36a6e94a517dd8c0a"
server_key_version: "encryptionkey@0"
I20260926 12:33:23.440729 11820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:23.455180 11820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:23.458978 11820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:23.459213 11820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:23.460793 11820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:23.460860 11820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:23.460914 11820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:23.460942 11820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:23.470727 11820 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:36619
I20260926 12:33:23.470857 11950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:36619 every 8 connection(s)
Sep 26 12:33:23 dist-test-slave-t7tk krb5kdc[11703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426003, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:23.471865 11820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:33:23.475817 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:23.461811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:48915 (local address 127.10.98.62:43177)
0926 12:33:23.461926 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:23.461929 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:23.463169 (+ 1240us) server_negotiation.cc:407] Connection header received
0926 12:33:23.464429 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:23.464433 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:23.464492 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:23.464587 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:23.466319 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.467175 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.468701 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.469079 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.469221 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:23.472016 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:23.472020 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:23.472022 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:23.472050 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:23.473682 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:23.474173 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:23.474176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:23.474177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:23.474257 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:23.474568 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:23.474572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:23.474574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:23.474763 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:23.474850 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:23.475509 (+ 659us) server_negotiation.cc:299] Negotiation successful
0926 12:33:23.475652 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 12:33:23.476784 11844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:23.462419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:43177 (local address 127.10.98.1:48915)
0926 12:33:23.462989 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:23.463043 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:23.464136 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:23.464819 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:23.464829 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:23.465393 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:23.466120 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:23.466143 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.467338 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.467342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:23.468530 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:23.468541 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.469265 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.469270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:23.469466 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:23.470061 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:23.470078 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:23.471823 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:23.473798 (+ 1975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:23.473804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:23.473820 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:23.474074 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:23.474340 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:23.474343 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:23.474345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:23.474486 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:23.474861 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:23.474869 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:23.475139 (+ 270us) client_negotiation.cc:769] Sending connection context
0926 12:33:23.475438 (+ 299us) client_negotiation.cc:240] Negotiation successful
0926 12:33:23.475808 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":132,"threads_started":1}
I20260926 12:33:23.478138 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 11820
I20260926 12:33:23.478245 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:33:23.478310 11951 heartbeater.cc:347] Connected to a master server at 127.10.98.62:43177
I20260926 12:33:23.478448 11951 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:23.478533 10632 external_mini_cluster.cc:1428] Setting key ad7188f6056d4544338b3358fc89207f
I20260926 12:33:23.478960 11951 heartbeater.cc:511] Master 127.10.98.62:43177 requested a full tablet report, sending...
I20260926 12:33:23.481277 11746 ts_manager.cc:194] Registered new tserver with Master: a774498f426c4efdadba4d25a124c4ae (127.10.98.1:36619)
I20260926 12:33:23.482607 11746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:48915
WARNING: no policy specified for kudu/127.10.98.2@KRBTEST.COM; defaulting to no policy
I20260926 12:33:23.488263 11957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:23.480596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:43177 (local address 127.10.98.1:36957)
0926 12:33:23.481293 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:23.481308 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:23.481412 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:23.481764 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:23.481767 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:23.481975 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:23.482116 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:23.482124 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.483124 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.483128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:23.483795 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:23.483805 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.484276 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.484280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:23.484371 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:23.484839 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:23.484850 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:23.485307 (+ 457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:23.486824 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:23.486827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:23.486830 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:23.487021 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:23.487271 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:23.487274 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:23.487276 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:23.487353 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:23.487778 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:23.487782 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:23.487882 (+ 100us) client_negotiation.cc:769] Sending connection context
0926 12:33:23.487988 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 12:33:23.488109 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":98,"threads_started":1}
I20260926 12:33:23.488309 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:23.480669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:36957 (local address 127.10.98.62:43177)
0926 12:33:23.480779 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:23.480783 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:23.481381 (+ 598us) server_negotiation.cc:407] Connection header received
0926 12:33:23.481522 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:23.481525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:23.481579 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:23.481627 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:23.482243 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.482983 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.483907 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.484162 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.484248 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:23.485431 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:23.485435 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:23.485438 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:23.485463 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:23.486725 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:23.487114 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:23.487118 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:23.487120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:23.487190 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:23.487429 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:23.487433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:23.487435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:23.487660 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:23.487757 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:23.488090 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 12:33:23.488196 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
I20260926 12:33:23.508858 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:43177
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:34981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:23.623828 11960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:23.624120 11960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:23.624174 11960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:23.628242 11960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:23.628320 11960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:23.628360 11960 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:33:23.628397 11960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:33:23.634092 11960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:34981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.98.62:43177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:23.635197 11960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:23.636281 11960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:23.646679 11966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:23.646771 11960 server_base.cc:1034] running on GCE node
W20260926 12:33:23.646695 11968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:23.646715 11965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:23.647401 11960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:23.648089 11960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:23.649256 11960 hybrid_clock.cc:648] HybridClock initialized: now 1790426003649235 us; error 34 us; skew 500 ppm
Sep 26 12:33:23 dist-test-slave-t7tk krb5kdc[11703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426003, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:23.652081 11960 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:33:23.653045 11960 webserver.cc:460] Webserver started at http://127.10.98.2:44921/ using document root <none> and password file <none>
I20260926 12:33:23.653662 11960 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:23.653733 11960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:23.653931 11960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:23.656136 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "795e2135fc9546f98cfa6a259b717fae"
format_stamp: "Formatted at 2026-09-26 12:33:23 on dist-test-slave-t7tk"
server_key: "7c812636e47ed20c434c49710547e8e0"
server_key_iv: "92b26bc1d2d834c638e915e67f01d11e"
server_key_version: "encryptionkey@0"
I20260926 12:33:23.656760 11960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "795e2135fc9546f98cfa6a259b717fae"
format_stamp: "Formatted at 2026-09-26 12:33:23 on dist-test-slave-t7tk"
server_key: "7c812636e47ed20c434c49710547e8e0"
server_key_iv: "92b26bc1d2d834c638e915e67f01d11e"
server_key_version: "encryptionkey@0"
I20260926 12:33:23.660686 11960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:33:23.663393 11975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:23.664708 11960 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:33:23.664826 11960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "795e2135fc9546f98cfa6a259b717fae"
format_stamp: "Formatted at 2026-09-26 12:33:23 on dist-test-slave-t7tk"
server_key: "7c812636e47ed20c434c49710547e8e0"
server_key_iv: "92b26bc1d2d834c638e915e67f01d11e"
server_key_version: "encryptionkey@0"
I20260926 12:33:23.664973 11960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:23.692770 11960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:23.696328 11960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:23.696501 11960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:23.698040 11960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:23.698089 11960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:23.698132 11960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:23.698161 11960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:33:23 dist-test-slave-t7tk krb5kdc[11703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426003, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:23.708362 11960 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:45441
I20260926 12:33:23.708659 12090 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:45441 every 8 connection(s)
I20260926 12:33:23.709492 11960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:33:23.711724 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:23.698950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:56909 (local address 127.10.98.62:43177)
0926 12:33:23.699072 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:23.699077 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:23.700230 (+ 1153us) server_negotiation.cc:407] Connection header received
0926 12:33:23.701367 (+ 1137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:23.701370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:23.701430 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:23.701515 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:23.703175 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.703839 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.705062 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.705397 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.705494 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:23.708044 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:23.708048 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:23.708051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:23.708078 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:23.709686 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:23.710252 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:23.710255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:23.710257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:23.710350 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:23.710660 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:23.710664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:23.710666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:23.710846 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:23.710930 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:23.711411 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 12:33:23.711569 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 12:33:23.712468 11984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:23.699541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:43177 (local address 127.10.98.2:56909)
0926 12:33:23.700023 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:23.700117 (+ 94us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:23.701146 (+ 1029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:23.701746 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:23.701756 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:23.702235 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:23.703002 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:23.703025 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.703956 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.703960 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:23.704938 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:23.704948 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.705569 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.705573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:23.705731 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:23.706282 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:23.706299 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:23.707872 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:23.709793 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:23.709798 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:23.709809 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:23.710127 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:23.710435 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:23.710438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:23.710440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:23.710575 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:23.710943 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:23.710949 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:23.711193 (+ 244us) client_negotiation.cc:769] Sending connection context
0926 12:33:23.711354 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 12:33:23.711674 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":125,"threads_started":1}
I20260926 12:33:23.713703 12091 heartbeater.cc:347] Connected to a master server at 127.10.98.62:43177
I20260926 12:33:23.713837 12091 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:23.714475 12091 heartbeater.cc:511] Master 127.10.98.62:43177 requested a full tablet report, sending...
I20260926 12:33:23.715628 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 11960
I20260926 12:33:23.715690 11746 ts_manager.cc:194] Registered new tserver with Master: 795e2135fc9546f98cfa6a259b717fae (127.10.98.2:45441)
I20260926 12:33:23.715739 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:33:23.716018 10632 external_mini_cluster.cc:1428] Setting key 56ab0c1cce54f8266966635b2f6dc2ca
I20260926 12:33:23.716421 11746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:56909
I20260926 12:33:23.723453 12097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:23.715963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:43177 (local address 127.10.98.2:44615)
0926 12:33:23.716534 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:23.716557 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:23.716693 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:23.716918 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:23.716920 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:23.717211 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:23.717333 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:23.717339 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.718039 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.718043 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:23.718605 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:23.718612 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.719098 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.719100 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:23.719175 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:23.719624 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:23.719637 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:23.719936 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:23.721740 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:23.721743 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:23.721744 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:23.722008 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:23.722314 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:23.722317 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:23.722318 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:23.722400 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:23.722910 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:23.722912 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:23.723043 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 12:33:23.723149 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 12:33:23.723296 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":97,"threads_started":1}
I20260926 12:33:23.723512 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:23.716036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:44615 (local address 127.10.98.62:43177)
0926 12:33:23.716142 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:23.716146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:23.716626 (+ 480us) server_negotiation.cc:407] Connection header received
0926 12:33:23.716787 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:23.716790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:23.716830 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:23.716893 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:23.717439 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.717924 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.718703 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.718984 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.719080 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:23.720060 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:23.720063 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:23.720066 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:23.720088 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:23.721615 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:23.722105 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:23.722109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:23.722110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:23.722200 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:23.722486 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:23.722490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:23.722491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:23.722776 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:23.722881 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:23.723228 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 12:33:23.723349 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.10.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
I20260926 12:33:23.744766 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:43177
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:34981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:23.853910 12100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:23.854171 12100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:23.854213 12100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:23.858232 12100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:23.858299 12100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:23.858337 12100 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:33:23.858369 12100 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:33:23.863692 12100 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:34981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.98.62:43177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:23.864770 12100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:23.865800 12100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:23.875313 12106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:23.875353 12105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:23.875332 12108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:23.875615 12100 server_base.cc:1034] running on GCE node
I20260926 12:33:23.876049 12100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:23.876689 12100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:23.877871 12100 hybrid_clock.cc:648] HybridClock initialized: now 1790426003877849 us; error 36 us; skew 500 ppm
Sep 26 12:33:23 dist-test-slave-t7tk krb5kdc[11703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426003, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:23.881043 12100 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:33:23.881918 12100 webserver.cc:460] Webserver started at http://127.10.98.3:43567/ using document root <none> and password file <none>
I20260926 12:33:23.882524 12100 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:23.882594 12100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:23.882791 12100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:23.884796 12100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "a4d4990920624d97bcb8a5cb7a66373e"
format_stamp: "Formatted at 2026-09-26 12:33:23 on dist-test-slave-t7tk"
server_key: "eecdb576aea004aeaed51149b173a84a"
server_key_iv: "2aebfdbe3859e0b81310ed9231f0fa24"
server_key_version: "encryptionkey@0"
I20260926 12:33:23.885355 12100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "a4d4990920624d97bcb8a5cb7a66373e"
format_stamp: "Formatted at 2026-09-26 12:33:23 on dist-test-slave-t7tk"
server_key: "eecdb576aea004aeaed51149b173a84a"
server_key_iv: "2aebfdbe3859e0b81310ed9231f0fa24"
server_key_version: "encryptionkey@0"
I20260926 12:33:23.888937 12100 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 12:33:23.891633 12115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:23.892966 12100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:33:23.893208 12100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "a4d4990920624d97bcb8a5cb7a66373e"
format_stamp: "Formatted at 2026-09-26 12:33:23 on dist-test-slave-t7tk"
server_key: "eecdb576aea004aeaed51149b173a84a"
server_key_iv: "2aebfdbe3859e0b81310ed9231f0fa24"
server_key_version: "encryptionkey@0"
I20260926 12:33:23.893322 12100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:23.915601 12100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:23.918862 12100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:23.919036 12100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:23.920459 12100 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:23.920514 12100 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:23.920569 12100 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:23.920594 12100 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:33:23 dist-test-slave-t7tk krb5kdc[11703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426003, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:23.931046 12100 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:43831
I20260926 12:33:23.931102 12230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:43831 every 8 connection(s)
I20260926 12:33:23.932003 12100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:33:23.935086 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:23.921531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:50409 (local address 127.10.98.62:43177)
0926 12:33:23.921668 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:23.921672 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:23.922776 (+ 1104us) server_negotiation.cc:407] Connection header received
0926 12:33:23.923905 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:23.923908 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:23.923969 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:23.924055 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:23.925765 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.926380 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.928101 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.928338 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.928425 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:23.931462 (+ 3037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:23.931466 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:23.931469 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:23.931502 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:23.933213 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:23.933706 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:23.933708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:23.933710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:23.933782 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:23.934101 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:23.934104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:23.934105 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:23.934288 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:23.934362 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:23.934786 (+ 424us) server_negotiation.cc:299] Negotiation successful
0926 12:33:23.934884 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 12:33:23.935925 12124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:23.922143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:43177 (local address 127.10.98.3:50409)
0926 12:33:23.922621 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:23.922664 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:23.923687 (+ 1023us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:23.924288 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:23.924299 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:23.924780 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:23.925563 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:23.925588 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.926505 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.926509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:23.927965 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:23.927978 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.928510 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.928514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:23.928726 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:23.929496 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:23.929519 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:23.931278 (+ 1759us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:23.933335 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:23.933340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:23.933351 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:23.933625 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:23.933847 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:23.933849 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:23.933850 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:23.933973 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:23.934365 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:23.934369 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:23.934601 (+ 232us) client_negotiation.cc:769] Sending connection context
0926 12:33:23.934749 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 12:33:23.934999 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":125,"threads_started":1}
I20260926 12:33:23.937086 12231 heartbeater.cc:347] Connected to a master server at 127.10.98.62:43177
I20260926 12:33:23.937245 12231 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:23.937723 12231 heartbeater.cc:511] Master 127.10.98.62:43177 requested a full tablet report, sending...
I20260926 12:33:23.938831 11746 ts_manager.cc:194] Registered new tserver with Master: a4d4990920624d97bcb8a5cb7a66373e (127.10.98.3:43831)
I20260926 12:33:23.939622 11746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:50409
I20260926 12:33:23.941367 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 12100
I20260926 12:33:23.941494 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:33:23.941805 10632 external_mini_cluster.cc:1428] Setting key c4e79f5c848a2e8484ff3b639b598260
I20260926 12:33:23.943804 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:33:23.946117 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:23.939155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:54121 (local address 127.10.98.62:43177)
0926 12:33:23.939284 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:23.939289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:23.939507 (+ 218us) server_negotiation.cc:407] Connection header received
0926 12:33:23.939589 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:23.939593 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:23.939639 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:23.939738 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:23.940201 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.940728 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.941711 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.941930 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.942027 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:23.943049 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:23.943053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:23.943055 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:23.943078 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:23.944368 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:23.944770 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:23.944774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:23.944775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:23.944879 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:23.945162 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:23.945165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:23.945167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:23.945477 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:23.945581 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:23.945891 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 12:33:23.945999 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:33:23.946286 12237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:23.939165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:43177 (local address 127.10.98.3:54121)
0926 12:33:23.939428 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:23.939442 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:23.939522 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:23.939757 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:23.939760 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:23.939963 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:23.940089 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:23.940097 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.940852 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.940855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:23.941604 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:23.941614 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.942029 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.942032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:23.942116 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:23.942578 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:23.942590 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:23.942914 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:23.944456 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:23.944459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:23.944461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:23.944668 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:23.944970 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:23.944972 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:23.944974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:23.945051 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:23.945598 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:23.945600 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:23.945705 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 12:33:23.945821 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 12:33:23.946087 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":100,"threads_started":1}
Sep 26 12:33:23 dist-test-slave-t7tk krb5kdc[11703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426003, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 12:33:23 dist-test-slave-t7tk krb5kdc[11703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426003, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:23.970377 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:23.962016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35472 (local address 127.10.98.62:43177)
0926 12:33:23.962148 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:23.962152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:23.962165 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:33:23.962206 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:23.962208 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:23.962246 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:23.962319 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:23.963014 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.963605 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.964482 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.964712 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.964801 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:23.967163 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:23.967166 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:23.967168 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:23.967196 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:23.968652 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:23.969030 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:23.969032 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:23.969034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:23.969149 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:23.969504 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:23.969506 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:23.969507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:23.969732 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:23.969832 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:23.970112 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 12:33:23.970215 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:33:23.973594 11746 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:33:23.976704 11746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:33:23.990943 12248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:23.985345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:43831 (local address 127.0.0.1:36014)
0926 12:33:23.985679 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:23.985738 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:23.985872 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:23.986605 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:23.986611 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:23.986632 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:23.986780 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:23.986787 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.988031 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.988034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:23.989170 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:23.989180 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.990438 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.990441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:23.990524 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:23.990553 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 12:33:23.990645 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 12:33:23.990812 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":75,"threads_started":1}
I20260926 12:33:23.990954 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:23.985655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36014 (local address 127.10.98.3:43831)
0926 12:33:23.986116 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:23.986129 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:23.986154 (+ 25us) server_negotiation.cc:407] Connection header received
0926 12:33:23.986234 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:23.986239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:23.986447 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:23.986615 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:23.986909 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.987941 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.989300 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.990113 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.990211 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:23.990383 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:23.990697 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 12:33:23.990798 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":116,"threads_started":1}
I20260926 12:33:23.991748 12251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:23.986049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:45441 (local address 127.0.0.1:56606)
0926 12:33:23.986852 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:23.986865 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:23.986941 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:23.987420 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:23.987423 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:23.987440 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:23.987559 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:23.987566 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.988989 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.988993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:23.990087 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:23.990096 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.991331 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.991334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:23.991428 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:23.991450 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:33:23.991524 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 12:33:23.991593 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260926 12:33:23.991813 12253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:23.986429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.10.98.2:45441)
0926 12:33:23.986989 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:23.987004 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:23.987026 (+ 22us) server_negotiation.cc:407] Connection header received
0926 12:33:23.987091 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:23.987096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:23.987282 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:23.987430 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:23.987678 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.988857 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.990207 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.991008 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.991111 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:23.991269 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:23.991582 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 12:33:23.991681 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":457,"thread_start_us":140,"threads_started":1}
I20260926 12:33:23.992051 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:23.986148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32872 (local address 127.10.98.1:36619)
0926 12:33:23.986404 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:23.986411 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:23.987055 (+ 644us) server_negotiation.cc:407] Connection header received
0926 12:33:23.987137 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:23.987143 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:23.987346 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:23.987478 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:23.987808 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.988904 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.990389 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.991142 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.991232 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:23.991355 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:23.991830 (+ 475us) server_negotiation.cc:299] Negotiation successful
0926 12:33:23.991935 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":83,"threads_started":1}
I20260926 12:33:23.991763 12249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:23.985369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:36619 (local address 127.0.0.1:32872)
0926 12:33:23.986478 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:23.986490 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:23.986585 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:23.987578 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:23.987581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:23.987598 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:23.987699 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:23.987705 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.989017 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.989020 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:23.990276 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:23.990285 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:23.991452 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:23.991455 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:23.991531 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:23.991549 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 12:33:23.991604 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 12:33:23.991666 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":52,"thread_start_us":88,"threads_started":1}
I20260926 12:33:23.994565 12165 tablet_service.cc:1467] Processing CreateTablet for tablet 22024cd1ffe445ceb669fb017f2ec8a0 (DEFAULT_TABLE table=test-table [id=fc104b97712949c6bdf695a18fa18c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:23.994688 11885 tablet_service.cc:1467] Processing CreateTablet for tablet 22024cd1ffe445ceb669fb017f2ec8a0 (DEFAULT_TABLE table=test-table [id=fc104b97712949c6bdf695a18fa18c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:23.995750 12165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22024cd1ffe445ceb669fb017f2ec8a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:23.995745 11885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22024cd1ffe445ceb669fb017f2ec8a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:23.999470 12025 tablet_service.cc:1467] Processing CreateTablet for tablet 22024cd1ffe445ceb669fb017f2ec8a0 (DEFAULT_TABLE table=test-table [id=fc104b97712949c6bdf695a18fa18c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:24.000370 12025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22024cd1ffe445ceb669fb017f2ec8a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:24.003072 12254 tablet_bootstrap.cc:492] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae: Bootstrap starting.
I20260926 12:33:24.004004 12255 tablet_bootstrap.cc:492] T 22024cd1ffe445ceb669fb017f2ec8a0 P a4d4990920624d97bcb8a5cb7a66373e: Bootstrap starting.
I20260926 12:33:24.005700 12254 tablet_bootstrap.cc:654] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:24.006068 12255 tablet_bootstrap.cc:654] T 22024cd1ffe445ceb669fb017f2ec8a0 P a4d4990920624d97bcb8a5cb7a66373e: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:24.006596 12254 log.cc:824] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:24.006726 12255 log.cc:824] T 22024cd1ffe445ceb669fb017f2ec8a0 P a4d4990920624d97bcb8a5cb7a66373e: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:24.008569 12254 tablet_bootstrap.cc:492] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae: No bootstrap required, opened a new log
I20260926 12:33:24.008723 12255 tablet_bootstrap.cc:492] T 22024cd1ffe445ceb669fb017f2ec8a0 P a4d4990920624d97bcb8a5cb7a66373e: No bootstrap required, opened a new log
I20260926 12:33:24.008906 12255 ts_tablet_manager.cc:1397] T 22024cd1ffe445ceb669fb017f2ec8a0 P a4d4990920624d97bcb8a5cb7a66373e: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 12:33:24.008904 12254 ts_tablet_manager.cc:1397] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:33:24.008899 12258 tablet_bootstrap.cc:492] T 22024cd1ffe445ceb669fb017f2ec8a0 P 795e2135fc9546f98cfa6a259b717fae: Bootstrap starting.
I20260926 12:33:24.011166 12258 tablet_bootstrap.cc:654] T 22024cd1ffe445ceb669fb017f2ec8a0 P 795e2135fc9546f98cfa6a259b717fae: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:24.011992 12258 log.cc:824] T 22024cd1ffe445ceb669fb017f2ec8a0 P 795e2135fc9546f98cfa6a259b717fae: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:24.011970 12255 raft_consensus.cc:348] T 22024cd1ffe445ceb669fb017f2ec8a0 P a4d4990920624d97bcb8a5cb7a66373e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } } peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } }
I20260926 12:33:24.012210 12255 raft_consensus.cc:374] T 22024cd1ffe445ceb669fb017f2ec8a0 P a4d4990920624d97bcb8a5cb7a66373e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:24.012264 12255 raft_consensus.cc:729] T 22024cd1ffe445ceb669fb017f2ec8a0 P a4d4990920624d97bcb8a5cb7a66373e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d4990920624d97bcb8a5cb7a66373e, State: Initialized, Role: FOLLOWER
I20260926 12:33:24.012676 12255 consensus_queue.cc:260] T 22024cd1ffe445ceb669fb017f2ec8a0 P a4d4990920624d97bcb8a5cb7a66373e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } } peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } }
I20260926 12:33:24.013355 12231 heartbeater.cc:503] Master 127.10.98.62:43177 was elected leader, sending a full tablet report...
I20260926 12:33:24.013233 12254 raft_consensus.cc:348] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } } peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } }
I20260926 12:33:24.013525 12254 raft_consensus.cc:374] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:24.013577 12254 raft_consensus.cc:729] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a774498f426c4efdadba4d25a124c4ae, State: Initialized, Role: FOLLOWER
I20260926 12:33:24.013679 12255 ts_tablet_manager.cc:1428] T 22024cd1ffe445ceb669fb017f2ec8a0 P a4d4990920624d97bcb8a5cb7a66373e: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260926 12:33:24.013989 12254 consensus_queue.cc:260] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } } peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } }
I20260926 12:33:24.014833 11951 heartbeater.cc:503] Master 127.10.98.62:43177 was elected leader, sending a full tablet report...
I20260926 12:33:24.015910 12254 ts_tablet_manager.cc:1428] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 12:33:24.017604 12258 tablet_bootstrap.cc:492] T 22024cd1ffe445ceb669fb017f2ec8a0 P 795e2135fc9546f98cfa6a259b717fae: No bootstrap required, opened a new log
I20260926 12:33:24.017810 12258 ts_tablet_manager.cc:1397] T 22024cd1ffe445ceb669fb017f2ec8a0 P 795e2135fc9546f98cfa6a259b717fae: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260926 12:33:24.020833 12258 raft_consensus.cc:348] T 22024cd1ffe445ceb669fb017f2ec8a0 P 795e2135fc9546f98cfa6a259b717fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } } peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } }
I20260926 12:33:24.021049 12258 raft_consensus.cc:374] T 22024cd1ffe445ceb669fb017f2ec8a0 P 795e2135fc9546f98cfa6a259b717fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:24.021080 12258 raft_consensus.cc:729] T 22024cd1ffe445ceb669fb017f2ec8a0 P 795e2135fc9546f98cfa6a259b717fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795e2135fc9546f98cfa6a259b717fae, State: Initialized, Role: FOLLOWER
I20260926 12:33:24.021552 12258 consensus_queue.cc:260] T 22024cd1ffe445ceb669fb017f2ec8a0 P 795e2135fc9546f98cfa6a259b717fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } } peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } }
I20260926 12:33:24.022277 12091 heartbeater.cc:503] Master 127.10.98.62:43177 was elected leader, sending a full tablet report...
I20260926 12:33:24.022462 12258 ts_tablet_manager.cc:1428] T 22024cd1ffe445ceb669fb017f2ec8a0 P 795e2135fc9546f98cfa6a259b717fae: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:33:24.064576 12261 raft_consensus.cc:482] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:33:24.064815 12261 raft_consensus.cc:504] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } } peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } }
I20260926 12:33:24.066201 12261 leader_election.cc:288] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a4d4990920624d97bcb8a5cb7a66373e (127.10.98.3:43831), 795e2135fc9546f98cfa6a259b717fae (127.10.98.2:45441)
I20260926 12:33:24.070410 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.066639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:55241 (local address 127.10.98.3:43831)
0926 12:33:24.066801 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:24.066807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:24.066983 (+ 176us) server_negotiation.cc:407] Connection header received
0926 12:33:24.067196 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:24.067200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:24.067263 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:24.067347 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:24.067654 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.068168 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.069436 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.069859 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.069965 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.070036 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:24.070200 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 12:33:24.070277 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 12:33:24.070436 12263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.066514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:43831 (local address 127.10.98.1:55241)
0926 12:33:24.066889 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:24.066902 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:24.067018 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:24.067387 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:24.067390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:24.067423 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:24.067541 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:24.067548 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.068284 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.068288 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:24.069297 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:24.069308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.069980 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.069984 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:24.070083 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.070104 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:33:24.070178 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 12:33:24.070252 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":37,"thread_start_us":148,"threads_started":1}
I20260926 12:33:24.070747 12253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.067172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:54635 (local address 127.10.98.2:45441)
0926 12:33:24.067308 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:24.067312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:24.067399 (+ 87us) server_negotiation.cc:407] Connection header received
0926 12:33:24.067588 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:24.067590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:24.067630 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:24.067689 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:24.067929 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.068604 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.069781 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.070197 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.070303 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.070360 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:24.070559 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 12:33:24.070636 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 12:33:24.071066 12264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.067023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:45441 (local address 127.10.98.1:54635)
0926 12:33:24.067309 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:24.067321 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:24.067479 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:24.067707 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:24.067709 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:24.067724 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:24.067837 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:24.067842 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.068712 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.068716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:24.069682 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:24.069693 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.070313 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.070317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:24.070410 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.070432 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:33:24.070881 (+ 449us) client_negotiation.cc:240] Negotiation successful
0926 12:33:24.070955 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":27,"spinlock_wait_cycles":20864,"thread_start_us":97,"threads_started":1}
I20260926 12:33:24.071108 12185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22024cd1ffe445ceb669fb017f2ec8a0" candidate_uuid: "a774498f426c4efdadba4d25a124c4ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4d4990920624d97bcb8a5cb7a66373e" is_pre_election: true
I20260926 12:33:24.071472 12185 raft_consensus.cc:2454] T 22024cd1ffe445ceb669fb017f2ec8a0 P a4d4990920624d97bcb8a5cb7a66373e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a774498f426c4efdadba4d25a124c4ae in term 0.
I20260926 12:33:24.071650 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22024cd1ffe445ceb669fb017f2ec8a0" candidate_uuid: "a774498f426c4efdadba4d25a124c4ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795e2135fc9546f98cfa6a259b717fae" is_pre_election: true
I20260926 12:33:24.071915 12045 raft_consensus.cc:2454] T 22024cd1ffe445ceb669fb017f2ec8a0 P 795e2135fc9546f98cfa6a259b717fae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a774498f426c4efdadba4d25a124c4ae in term 0.
I20260926 12:33:24.072028 11839 leader_election.cc:302] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4d4990920624d97bcb8a5cb7a66373e, a774498f426c4efdadba4d25a124c4ae; no voters:
I20260926 12:33:24.072320 12261 raft_consensus.cc:2781] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:33:24.072403 12261 raft_consensus.cc:482] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:33:24.072449 12261 raft_consensus.cc:3037] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:24.073663 12261 raft_consensus.cc:504] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } } peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } }
I20260926 12:33:24.073987 12261 leader_election.cc:288] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae [CANDIDATE]: Term 1 election: Requested vote from peers a4d4990920624d97bcb8a5cb7a66373e (127.10.98.3:43831), 795e2135fc9546f98cfa6a259b717fae (127.10.98.2:45441)
I20260926 12:33:24.074299 12185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22024cd1ffe445ceb669fb017f2ec8a0" candidate_uuid: "a774498f426c4efdadba4d25a124c4ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4d4990920624d97bcb8a5cb7a66373e"
I20260926 12:33:24.074369 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22024cd1ffe445ceb669fb017f2ec8a0" candidate_uuid: "a774498f426c4efdadba4d25a124c4ae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795e2135fc9546f98cfa6a259b717fae"
I20260926 12:33:24.074406 12185 raft_consensus.cc:3037] T 22024cd1ffe445ceb669fb017f2ec8a0 P a4d4990920624d97bcb8a5cb7a66373e [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:24.074493 12045 raft_consensus.cc:3037] T 22024cd1ffe445ceb669fb017f2ec8a0 P 795e2135fc9546f98cfa6a259b717fae [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:24.075601 12185 raft_consensus.cc:2454] T 22024cd1ffe445ceb669fb017f2ec8a0 P a4d4990920624d97bcb8a5cb7a66373e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a774498f426c4efdadba4d25a124c4ae in term 1.
I20260926 12:33:24.075863 12045 raft_consensus.cc:2454] T 22024cd1ffe445ceb669fb017f2ec8a0 P 795e2135fc9546f98cfa6a259b717fae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a774498f426c4efdadba4d25a124c4ae in term 1.
I20260926 12:33:24.075873 11839 leader_election.cc:302] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4d4990920624d97bcb8a5cb7a66373e, a774498f426c4efdadba4d25a124c4ae; no voters:
I20260926 12:33:24.076025 12261 raft_consensus.cc:2781] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:24.076263 12261 raft_consensus.cc:686] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae [term 1 LEADER]: Becoming Leader. State: Replica: a774498f426c4efdadba4d25a124c4ae, State: Running, Role: LEADER
I20260926 12:33:24.076647 12261 consensus_queue.cc:237] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } } peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } }
I20260926 12:33:24.079967 11748 catalog_manager.cc:5495] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae reported cstate change: term changed from 0 to 1, leader changed from <none> to a774498f426c4efdadba4d25a124c4ae (127.10.98.1). New cstate: current_term: 1 leader_uuid: "a774498f426c4efdadba4d25a124c4ae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } health_report { overall_health: UNKNOWN } } }
I20260926 12:33:24.119468 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.115800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32880 (local address 127.10.98.1:36619)
0926 12:33:24.115930 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:24.115935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:24.115949 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:33:24.116143 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:24.116145 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:24.116203 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:24.116294 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:24.116538 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.117071 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.118089 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.118303 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.118383 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.118711 (+ 328us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:24.119121 (+ 410us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:24.119202 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:24.119298 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 12:33:24.119349 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:33:24.125319 12185 raft_consensus.cc:1261] T 22024cd1ffe445ceb669fb017f2ec8a0 P a4d4990920624d97bcb8a5cb7a66373e [term 1 FOLLOWER]: Refusing update from remote peer a774498f426c4efdadba4d25a124c4ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:24.125317 12045 raft_consensus.cc:1261] T 22024cd1ffe445ceb669fb017f2ec8a0 P 795e2135fc9546f98cfa6a259b717fae [term 1 FOLLOWER]: Refusing update from remote peer a774498f426c4efdadba4d25a124c4ae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:24.126048 12261 consensus_queue.cc:1035] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:24.126281 12265 consensus_queue.cc:1035] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:24.133131 12268 mvcc.cc:205] Tried to move back new op lower bound from 7333584912890269696 to 7333584912700858368. Current Snapshot: MvccSnapshot[applied={T|T < 7333584912890269696}]
I20260926 12:33:24.134724 12269 mvcc.cc:205] Tried to move back new op lower bound from 7333584912890269696 to 7333584912700858368. Current Snapshot: MvccSnapshot[applied={T|T < 7333584912890269696}]
I20260926 12:33:24.134980 12270 mvcc.cc:205] Tried to move back new op lower bound from 7333584912890269696 to 7333584912700858368. Current Snapshot: MvccSnapshot[applied={T|T < 7333584912890269696}]
I20260926 12:33:24.146165 11746 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:24.146330 11746 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:24.148723 11746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4: Sending DeleteTablet for 3 replicas of tablet 22024cd1ffe445ceb669fb017f2ec8a0
I20260926 12:33:24.149574 11885 tablet_service.cc:1514] Processing DeleteTablet for tablet 22024cd1ffe445ceb669fb017f2ec8a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:24 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:32872
I20260926 12:33:24.149637 12165 tablet_service.cc:1514] Processing DeleteTablet for tablet 22024cd1ffe445ceb669fb017f2ec8a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:24 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:36014
I20260926 12:33:24.149814 12025 tablet_service.cc:1514] Processing DeleteTablet for tablet 22024cd1ffe445ceb669fb017f2ec8a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:24 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:56606
I20260926 12:33:24.150349 12279 tablet_replica.cc:330] T 22024cd1ffe445ceb669fb017f2ec8a0 P 795e2135fc9546f98cfa6a259b717fae: stopping tablet replica
I20260926 12:33:24.150532 12277 tablet_replica.cc:330] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae: stopping tablet replica
I20260926 12:33:24.150761 12279 raft_consensus.cc:2229] T 22024cd1ffe445ceb669fb017f2ec8a0 P 795e2135fc9546f98cfa6a259b717fae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:24.150827 11748 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 12:33:24.150934 12277 raft_consensus.cc:2229] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:33:24.151139 12279 raft_consensus.cc:2258] T 22024cd1ffe445ceb669fb017f2ec8a0 P 795e2135fc9546f98cfa6a259b717fae [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 12:33:24.151257 11748 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:33:24.151324 12277 raft_consensus.cc:2258] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:24.151679 12278 tablet_replica.cc:330] T 22024cd1ffe445ceb669fb017f2ec8a0 P a4d4990920624d97bcb8a5cb7a66373e: stopping tablet replica
I20260926 12:33:24.152026 12278 raft_consensus.cc:2229] T 22024cd1ffe445ceb669fb017f2ec8a0 P a4d4990920624d97bcb8a5cb7a66373e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:24.152336 12278 raft_consensus.cc:2258] T 22024cd1ffe445ceb669fb017f2ec8a0 P a4d4990920624d97bcb8a5cb7a66373e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:24.152853 12279 ts_tablet_manager.cc:1905] T 22024cd1ffe445ceb669fb017f2ec8a0 P 795e2135fc9546f98cfa6a259b717fae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:24.152987 12277 ts_tablet_manager.cc:1905] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:24.155731 12279 ts_tablet_manager.cc:1918] T 22024cd1ffe445ceb669fb017f2ec8a0 P 795e2135fc9546f98cfa6a259b717fae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:33:24.155793 12277 ts_tablet_manager.cc:1918] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:33:24.155941 12277 log.cc:1196] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-0/wal/wals/22024cd1ffe445ceb669fb017f2ec8a0
I20260926 12:33:24.155939 12279 log.cc:1196] T 22024cd1ffe445ceb669fb017f2ec8a0 P 795e2135fc9546f98cfa6a259b717fae: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/22024cd1ffe445ceb669fb017f2ec8a0
I20260926 12:33:24.156260 12279 ts_tablet_manager.cc:1939] T 22024cd1ffe445ceb669fb017f2ec8a0 P 795e2135fc9546f98cfa6a259b717fae: Deleting consensus metadata
I20260926 12:33:24.156373 12277 ts_tablet_manager.cc:1939] T 22024cd1ffe445ceb669fb017f2ec8a0 P a774498f426c4efdadba4d25a124c4ae: Deleting consensus metadata
I20260926 12:33:24.157647 12025 tablet_service.cc:1467] Processing CreateTablet for tablet d39720085d8a411fb9e4858c539e5083 (DEFAULT_TABLE table=test-table [id=17b4346399f743e19a634957b2605e3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:24.157883 12025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d39720085d8a411fb9e4858c539e5083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:24.158604 12278 ts_tablet_manager.cc:1905] T 22024cd1ffe445ceb669fb017f2ec8a0 P a4d4990920624d97bcb8a5cb7a66373e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:24.158596 11733 catalog_manager.cc:4841] TS a774498f426c4efdadba4d25a124c4ae (127.10.98.1:36619): tablet 22024cd1ffe445ceb669fb017f2ec8a0 (table test-table [id=fc104b97712949c6bdf695a18fa18c97]) successfully deleted
I20260926 12:33:24.159741 12258 tablet_bootstrap.cc:492] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae: Bootstrap starting.
I20260926 12:33:24.160123 12165 tablet_service.cc:1467] Processing CreateTablet for tablet d39720085d8a411fb9e4858c539e5083 (DEFAULT_TABLE table=test-table [id=17b4346399f743e19a634957b2605e3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:24.160375 12165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d39720085d8a411fb9e4858c539e5083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:24.160696 12258 tablet_bootstrap.cc:654] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:24.161361 12278 ts_tablet_manager.cc:1918] T 22024cd1ffe445ceb669fb017f2ec8a0 P a4d4990920624d97bcb8a5cb7a66373e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:33:24.161491 12278 log.cc:1196] T 22024cd1ffe445ceb669fb017f2ec8a0 P a4d4990920624d97bcb8a5cb7a66373e: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/22024cd1ffe445ceb669fb017f2ec8a0
I20260926 12:33:24.161799 12278 ts_tablet_manager.cc:1939] T 22024cd1ffe445ceb669fb017f2ec8a0 P a4d4990920624d97bcb8a5cb7a66373e: Deleting consensus metadata
I20260926 12:33:24.162437 12255 tablet_bootstrap.cc:492] T d39720085d8a411fb9e4858c539e5083 P a4d4990920624d97bcb8a5cb7a66373e: Bootstrap starting.
I20260926 12:33:24.162686 11735 catalog_manager.cc:4841] TS 795e2135fc9546f98cfa6a259b717fae (127.10.98.2:45441): tablet 22024cd1ffe445ceb669fb017f2ec8a0 (table test-table [id=fc104b97712949c6bdf695a18fa18c97]) successfully deleted
I20260926 12:33:24.163110 11735 catalog_manager.cc:4841] TS a4d4990920624d97bcb8a5cb7a66373e (127.10.98.3:43831): tablet 22024cd1ffe445ceb669fb017f2ec8a0 (table test-table [id=fc104b97712949c6bdf695a18fa18c97]) successfully deleted
I20260926 12:33:24.163336 12255 tablet_bootstrap.cc:654] T d39720085d8a411fb9e4858c539e5083 P a4d4990920624d97bcb8a5cb7a66373e: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:24.163292 11885 tablet_service.cc:1467] Processing CreateTablet for tablet d39720085d8a411fb9e4858c539e5083 (DEFAULT_TABLE table=test-table [id=17b4346399f743e19a634957b2605e3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:24.163545 11885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d39720085d8a411fb9e4858c539e5083. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:24.164372 12255 tablet_bootstrap.cc:492] T d39720085d8a411fb9e4858c539e5083 P a4d4990920624d97bcb8a5cb7a66373e: No bootstrap required, opened a new log
I20260926 12:33:24.164453 12255 ts_tablet_manager.cc:1397] T d39720085d8a411fb9e4858c539e5083 P a4d4990920624d97bcb8a5cb7a66373e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 12:33:24.164837 12255 raft_consensus.cc:348] T d39720085d8a411fb9e4858c539e5083 P a4d4990920624d97bcb8a5cb7a66373e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } }
I20260926 12:33:24.164932 12255 raft_consensus.cc:374] T d39720085d8a411fb9e4858c539e5083 P a4d4990920624d97bcb8a5cb7a66373e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:24.164961 12255 raft_consensus.cc:729] T d39720085d8a411fb9e4858c539e5083 P a4d4990920624d97bcb8a5cb7a66373e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d4990920624d97bcb8a5cb7a66373e, State: Initialized, Role: FOLLOWER
I20260926 12:33:24.165122 12255 consensus_queue.cc:260] T d39720085d8a411fb9e4858c539e5083 P a4d4990920624d97bcb8a5cb7a66373e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } }
I20260926 12:33:24.165372 12255 ts_tablet_manager.cc:1428] T d39720085d8a411fb9e4858c539e5083 P a4d4990920624d97bcb8a5cb7a66373e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:24.165879 12254 tablet_bootstrap.cc:492] T d39720085d8a411fb9e4858c539e5083 P a774498f426c4efdadba4d25a124c4ae: Bootstrap starting.
I20260926 12:33:24.166859 12254 tablet_bootstrap.cc:654] T d39720085d8a411fb9e4858c539e5083 P a774498f426c4efdadba4d25a124c4ae: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:24.167198 12258 tablet_bootstrap.cc:492] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae: No bootstrap required, opened a new log
I20260926 12:33:24.167281 12258 ts_tablet_manager.cc:1397] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260926 12:33:24.167712 12258 raft_consensus.cc:348] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } }
I20260926 12:33:24.167831 12258 raft_consensus.cc:374] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:24.167860 12258 raft_consensus.cc:729] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795e2135fc9546f98cfa6a259b717fae, State: Initialized, Role: FOLLOWER
I20260926 12:33:24.167990 12258 consensus_queue.cc:260] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } }
I20260926 12:33:24.168233 12258 ts_tablet_manager.cc:1428] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 12:33:24.170356 12254 tablet_bootstrap.cc:492] T d39720085d8a411fb9e4858c539e5083 P a774498f426c4efdadba4d25a124c4ae: No bootstrap required, opened a new log
I20260926 12:33:24.170442 12254 ts_tablet_manager.cc:1397] T d39720085d8a411fb9e4858c539e5083 P a774498f426c4efdadba4d25a124c4ae: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 12:33:24.170912 12254 raft_consensus.cc:348] T d39720085d8a411fb9e4858c539e5083 P a774498f426c4efdadba4d25a124c4ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } }
I20260926 12:33:24.171015 12254 raft_consensus.cc:374] T d39720085d8a411fb9e4858c539e5083 P a774498f426c4efdadba4d25a124c4ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:24.171042 12254 raft_consensus.cc:729] T d39720085d8a411fb9e4858c539e5083 P a774498f426c4efdadba4d25a124c4ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a774498f426c4efdadba4d25a124c4ae, State: Initialized, Role: FOLLOWER
I20260926 12:33:24.171164 12254 consensus_queue.cc:260] T d39720085d8a411fb9e4858c539e5083 P a774498f426c4efdadba4d25a124c4ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } }
I20260926 12:33:24.171396 12254 ts_tablet_manager.cc:1428] T d39720085d8a411fb9e4858c539e5083 P a774498f426c4efdadba4d25a124c4ae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260926 12:33:24.182950 12232 tablet.cc:2369] T d39720085d8a411fb9e4858c539e5083 P a4d4990920624d97bcb8a5cb7a66373e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:33:24.210840 12092 tablet.cc:2369] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:33:24.224078 11952 tablet.cc:2369] T d39720085d8a411fb9e4858c539e5083 P a774498f426c4efdadba4d25a124c4ae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:33:24.287469 12262 raft_consensus.cc:482] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:33:24.287631 12262 raft_consensus.cc:504] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } }
I20260926 12:33:24.288723 12262 leader_election.cc:288] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a774498f426c4efdadba4d25a124c4ae (127.10.98.1:36619), a4d4990920624d97bcb8a5cb7a66373e (127.10.98.3:43831)
I20260926 12:33:24.293259 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.289205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:34365 (local address 127.10.98.1:36619)
0926 12:33:24.289336 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:24.289340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:24.289456 (+ 116us) server_negotiation.cc:407] Connection header received
0926 12:33:24.289678 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:24.289681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:24.289721 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:24.289757 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:24.290086 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.290872 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.292042 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.292642 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.292737 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.292816 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:24.293012 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 12:33:24.293127 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:33:24.293920 12283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.288954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:36619 (local address 127.10.98.2:34365)
0926 12:33:24.289374 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:24.289389 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:24.289547 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:24.289885 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:24.289888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:24.289900 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:24.289977 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:24.289982 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.291012 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.291016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:24.291921 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:24.291931 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.292762 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.292766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:24.292862 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.292890 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 12:33:24.293653 (+ 763us) client_negotiation.cc:240] Negotiation successful
0926 12:33:24.293774 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":1,"thread_start_us":179,"threads_started":1}
I20260926 12:33:24.294175 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.289374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:51339 (local address 127.10.98.3:43831)
0926 12:33:24.289494 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:24.289498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:24.289566 (+ 68us) server_negotiation.cc:407] Connection header received
0926 12:33:24.289663 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:24.289666 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:24.289713 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:24.289790 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:24.291031 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.291760 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.292800 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.293374 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.293454 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.293511 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:24.293994 (+ 483us) server_negotiation.cc:299] Negotiation successful
0926 12:33:24.294070 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:33:24.294493 12284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.289074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:43831 (local address 127.10.98.2:51339)
0926 12:33:24.289474 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:24.289485 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:24.289576 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:24.289813 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:24.289816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:24.289834 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:24.289955 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:24.289963 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.291875 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.291879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:24.292666 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:24.292676 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.293680 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.293684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:24.293854 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.293876 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:33:24.294296 (+ 420us) client_negotiation.cc:240] Negotiation successful
0926 12:33:24.294368 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"mutex_wait_us":63,"thread_start_us":74,"threads_started":1}
I20260926 12:33:24.294674 11905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39720085d8a411fb9e4858c539e5083" candidate_uuid: "795e2135fc9546f98cfa6a259b717fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a774498f426c4efdadba4d25a124c4ae" is_pre_election: true
I20260926 12:33:24.294924 11905 raft_consensus.cc:2454] T d39720085d8a411fb9e4858c539e5083 P a774498f426c4efdadba4d25a124c4ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795e2135fc9546f98cfa6a259b717fae in term 0.
I20260926 12:33:24.295030 12185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39720085d8a411fb9e4858c539e5083" candidate_uuid: "795e2135fc9546f98cfa6a259b717fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4d4990920624d97bcb8a5cb7a66373e" is_pre_election: true
I20260926 12:33:24.295159 12185 raft_consensus.cc:2454] T d39720085d8a411fb9e4858c539e5083 P a4d4990920624d97bcb8a5cb7a66373e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 795e2135fc9546f98cfa6a259b717fae in term 0.
I20260926 12:33:24.295356 11977 leader_election.cc:302] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795e2135fc9546f98cfa6a259b717fae, a774498f426c4efdadba4d25a124c4ae; no voters:
I20260926 12:33:24.295604 12262 raft_consensus.cc:2781] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:33:24.295660 12262 raft_consensus.cc:482] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:33:24.295696 12262 raft_consensus.cc:3037] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:24.296705 12262 raft_consensus.cc:504] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } }
I20260926 12:33:24.297039 12262 leader_election.cc:288] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae [CANDIDATE]: Term 1 election: Requested vote from peers a774498f426c4efdadba4d25a124c4ae (127.10.98.1:36619), a4d4990920624d97bcb8a5cb7a66373e (127.10.98.3:43831)
I20260926 12:33:24.297338 11905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39720085d8a411fb9e4858c539e5083" candidate_uuid: "795e2135fc9546f98cfa6a259b717fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a774498f426c4efdadba4d25a124c4ae"
I20260926 12:33:24.297473 11905 raft_consensus.cc:3037] T d39720085d8a411fb9e4858c539e5083 P a774498f426c4efdadba4d25a124c4ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:24.297448 12185 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d39720085d8a411fb9e4858c539e5083" candidate_uuid: "795e2135fc9546f98cfa6a259b717fae" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4d4990920624d97bcb8a5cb7a66373e"
I20260926 12:33:24.297554 12185 raft_consensus.cc:3037] T d39720085d8a411fb9e4858c539e5083 P a4d4990920624d97bcb8a5cb7a66373e [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:24.298532 11905 raft_consensus.cc:2454] T d39720085d8a411fb9e4858c539e5083 P a774498f426c4efdadba4d25a124c4ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795e2135fc9546f98cfa6a259b717fae in term 1.
I20260926 12:33:24.298532 12185 raft_consensus.cc:2454] T d39720085d8a411fb9e4858c539e5083 P a4d4990920624d97bcb8a5cb7a66373e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 795e2135fc9546f98cfa6a259b717fae in term 1.
I20260926 12:33:24.298791 11977 leader_election.cc:302] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795e2135fc9546f98cfa6a259b717fae, a774498f426c4efdadba4d25a124c4ae; no voters:
I20260926 12:33:24.298957 12262 raft_consensus.cc:2781] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:24.299146 12262 raft_consensus.cc:686] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae [term 1 LEADER]: Becoming Leader. State: Replica: 795e2135fc9546f98cfa6a259b717fae, State: Running, Role: LEADER
I20260926 12:33:24.299379 12262 consensus_queue.cc:237] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } }
I20260926 12:33:24.301512 11746 catalog_manager.cc:5495] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae reported cstate change: term changed from 0 to 1, leader changed from <none> to 795e2135fc9546f98cfa6a259b717fae (127.10.98.2). New cstate: current_term: 1 leader_uuid: "795e2135fc9546f98cfa6a259b717fae" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } health_report { overall_health: UNKNOWN } } }
Sep 26 12:33:24 dist-test-slave-t7tk krb5kdc[11703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426003, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:24.355917 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.346960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35486 (local address 127.10.98.62:43177)
0926 12:33:24.347106 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:24.347111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:24.347475 (+ 364us) server_negotiation.cc:407] Connection header received
0926 12:33:24.347541 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:24.347543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:24.347585 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:24.347668 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:24.348430 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.348967 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.349834 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.350073 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.350168 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.352557 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:24.352561 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:24.352562 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:24.352585 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:24.354138 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:24.354619 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:24.354622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:24.354624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:24.354703 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:24.354975 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:24.354978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:24.354980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:24.355170 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:24.355266 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:24.355579 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 12:33:24.355771 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":79,"server-negotiator.queue_time_us":61}
I20260926 12:33:24.355918 12292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.346896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:43177 (local address 127.0.0.1:35486)
0926 12:33:24.347316 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:24.347329 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:24.347441 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:24.347729 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:24.347732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:24.348206 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:24.348304 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:24.348310 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.349088 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.349091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:24.349703 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:24.349712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.350205 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.350208 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:24.350274 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.350867 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:24.350880 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:24.352409 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:24.354248 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:24.354261 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:24.354269 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:24.354521 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:24.354793 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:24.354796 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:24.354798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:24.354891 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:24.355276 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:24.355284 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:24.355407 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 12:33:24.355518 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 12:33:24.355694 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":191,"threads_started":1}
I20260926 12:33:24.360136 11746 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:35486:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 12:33:24.360875 11746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:33:24.365561 11885 tablet_service.cc:1467] Processing CreateTablet for tablet b4f2a0e173a04636adba7fdf460b0073 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=43f03be127ce410ab86dd2d762fece1e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:33:24.365720 12165 tablet_service.cc:1467] Processing CreateTablet for tablet b4f2a0e173a04636adba7fdf460b0073 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=43f03be127ce410ab86dd2d762fece1e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:33:24.365774 12025 tablet_service.cc:1467] Processing CreateTablet for tablet b4f2a0e173a04636adba7fdf460b0073 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=43f03be127ce410ab86dd2d762fece1e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:33:24.366077 12165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f2a0e173a04636adba7fdf460b0073. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:24.366204 12025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f2a0e173a04636adba7fdf460b0073. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:24.366204 11885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b4f2a0e173a04636adba7fdf460b0073. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:24.368837 12255 tablet_bootstrap.cc:492] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e: Bootstrap starting.
I20260926 12:33:24.368920 12254 tablet_bootstrap.cc:492] T b4f2a0e173a04636adba7fdf460b0073 P a774498f426c4efdadba4d25a124c4ae: Bootstrap starting.
I20260926 12:33:24.368961 12258 tablet_bootstrap.cc:492] T b4f2a0e173a04636adba7fdf460b0073 P 795e2135fc9546f98cfa6a259b717fae: Bootstrap starting.
I20260926 12:33:24.369966 12254 tablet_bootstrap.cc:654] T b4f2a0e173a04636adba7fdf460b0073 P a774498f426c4efdadba4d25a124c4ae: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:24.369966 12258 tablet_bootstrap.cc:654] T b4f2a0e173a04636adba7fdf460b0073 P 795e2135fc9546f98cfa6a259b717fae: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:24.369987 12255 tablet_bootstrap.cc:654] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:24.371102 12255 tablet_bootstrap.cc:492] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e: No bootstrap required, opened a new log
I20260926 12:33:24.371191 12255 ts_tablet_manager.cc:1397] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 12:33:24.371428 12258 tablet_bootstrap.cc:492] T b4f2a0e173a04636adba7fdf460b0073 P 795e2135fc9546f98cfa6a259b717fae: No bootstrap required, opened a new log
I20260926 12:33:24.371515 12258 ts_tablet_manager.cc:1397] T b4f2a0e173a04636adba7fdf460b0073 P 795e2135fc9546f98cfa6a259b717fae: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:33:24.371639 12255 raft_consensus.cc:348] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } }
I20260926 12:33:24.371732 12254 tablet_bootstrap.cc:492] T b4f2a0e173a04636adba7fdf460b0073 P a774498f426c4efdadba4d25a124c4ae: No bootstrap required, opened a new log
I20260926 12:33:24.371760 12255 raft_consensus.cc:374] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:24.371790 12255 raft_consensus.cc:729] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4d4990920624d97bcb8a5cb7a66373e, State: Initialized, Role: FOLLOWER
I20260926 12:33:24.371803 12254 ts_tablet_manager.cc:1397] T b4f2a0e173a04636adba7fdf460b0073 P a774498f426c4efdadba4d25a124c4ae: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:33:24.371899 12258 raft_consensus.cc:348] T b4f2a0e173a04636adba7fdf460b0073 P 795e2135fc9546f98cfa6a259b717fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } }
I20260926 12:33:24.371934 12255 consensus_queue.cc:260] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } }
I20260926 12:33:24.372000 12258 raft_consensus.cc:374] T b4f2a0e173a04636adba7fdf460b0073 P 795e2135fc9546f98cfa6a259b717fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:24.372022 12258 raft_consensus.cc:729] T b4f2a0e173a04636adba7fdf460b0073 P 795e2135fc9546f98cfa6a259b717fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795e2135fc9546f98cfa6a259b717fae, State: Initialized, Role: FOLLOWER
I20260926 12:33:24.372125 12258 consensus_queue.cc:260] T b4f2a0e173a04636adba7fdf460b0073 P 795e2135fc9546f98cfa6a259b717fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } }
I20260926 12:33:24.372180 12254 raft_consensus.cc:348] T b4f2a0e173a04636adba7fdf460b0073 P a774498f426c4efdadba4d25a124c4ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } }
I20260926 12:33:24.372275 12254 raft_consensus.cc:374] T b4f2a0e173a04636adba7fdf460b0073 P a774498f426c4efdadba4d25a124c4ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:24.372205 12260 tablet_replica.cc:439] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } } }
I20260926 12:33:24.372304 12254 raft_consensus.cc:729] T b4f2a0e173a04636adba7fdf460b0073 P a774498f426c4efdadba4d25a124c4ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a774498f426c4efdadba4d25a124c4ae, State: Initialized, Role: FOLLOWER
I20260926 12:33:24.372359 12260 tablet_replica.cc:442] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:33:24.372390 12258 ts_tablet_manager.cc:1428] T b4f2a0e173a04636adba7fdf460b0073 P 795e2135fc9546f98cfa6a259b717fae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:24.372351 12262 tablet_replica.cc:439] T b4f2a0e173a04636adba7fdf460b0073 P 795e2135fc9546f98cfa6a259b717fae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } } }
I20260926 12:33:24.372412 12254 consensus_queue.cc:260] T b4f2a0e173a04636adba7fdf460b0073 P a774498f426c4efdadba4d25a124c4ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } }
I20260926 12:33:24.372465 12262 tablet_replica.cc:442] T b4f2a0e173a04636adba7fdf460b0073 P 795e2135fc9546f98cfa6a259b717fae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:33:24.372668 12254 ts_tablet_manager.cc:1428] T b4f2a0e173a04636adba7fdf460b0073 P a774498f426c4efdadba4d25a124c4ae: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:24.372637 12261 tablet_replica.cc:439] T b4f2a0e173a04636adba7fdf460b0073 P a774498f426c4efdadba4d25a124c4ae: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } } }
I20260926 12:33:24.372748 12261 tablet_replica.cc:442] T b4f2a0e173a04636adba7fdf460b0073 P a774498f426c4efdadba4d25a124c4ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:33:24.373411 12255 ts_tablet_manager.cc:1428] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 12:33:24.471562 12260 raft_consensus.cc:482] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:33:24.471714 12260 raft_consensus.cc:504] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } }
I20260926 12:33:24.472995 12260 leader_election.cc:288] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a774498f426c4efdadba4d25a124c4ae (127.10.98.1:36619), 795e2135fc9546f98cfa6a259b717fae (127.10.98.2:45441)
I20260926 12:33:24.477829 12253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.473386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:47231 (local address 127.10.98.2:45441)
0926 12:33:24.473521 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:24.473526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:24.473763 (+ 237us) server_negotiation.cc:407] Connection header received
0926 12:33:24.473973 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:24.473977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:24.474028 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:24.474106 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:24.474629 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.475444 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.476700 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.477225 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.477331 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.477399 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:24.477624 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 12:33:24.477703 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:33:24.477846 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.473332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:45441 (local address 127.10.98.3:47231)
0926 12:33:24.473677 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:24.473691 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:24.473836 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:24.474395 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:24.474398 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:24.474414 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:24.474533 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:24.474539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.475560 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.475564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:24.476537 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:24.476547 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.477376 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.477379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:24.477497 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.477524 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 12:33:24.477593 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 12:33:24.477669 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":70,"thread_start_us":92,"threads_started":1}
I20260926 12:33:24.478166 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.473371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:34469 (local address 127.10.98.1:36619)
0926 12:33:24.473530 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:24.473535 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:24.473946 (+ 411us) server_negotiation.cc:407] Connection header received
0926 12:33:24.474170 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:24.474174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:24.474219 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:24.474292 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:24.474629 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.475409 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.477017 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.477558 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.477663 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.477721 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:24.478004 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 12:33:24.478080 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260926 12:33:24.478550 12297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.473615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:36619 (local address 127.10.98.3:34469)
0926 12:33:24.473833 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:24.473845 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:24.474016 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:24.474340 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:24.474343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:24.474360 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:24.474461 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:24.474466 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.475536 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.475539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:24.476880 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:24.476891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.477780 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.477783 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:24.477871 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.477893 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:33:24.478341 (+ 448us) client_negotiation.cc:240] Negotiation successful
0926 12:33:24.478422 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":29,"thread_start_us":49,"threads_started":1}
I20260926 12:33:24.478488 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f2a0e173a04636adba7fdf460b0073" candidate_uuid: "a4d4990920624d97bcb8a5cb7a66373e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795e2135fc9546f98cfa6a259b717fae" is_pre_election: true
I20260926 12:33:24.478673 12045 raft_consensus.cc:2454] T b4f2a0e173a04636adba7fdf460b0073 P 795e2135fc9546f98cfa6a259b717fae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4d4990920624d97bcb8a5cb7a66373e in term 0.
I20260926 12:33:24.478955 11905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f2a0e173a04636adba7fdf460b0073" candidate_uuid: "a4d4990920624d97bcb8a5cb7a66373e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a774498f426c4efdadba4d25a124c4ae" is_pre_election: true
I20260926 12:33:24.479104 11905 raft_consensus.cc:2454] T b4f2a0e173a04636adba7fdf460b0073 P a774498f426c4efdadba4d25a124c4ae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4d4990920624d97bcb8a5cb7a66373e in term 0.
I20260926 12:33:24.479157 12119 leader_election.cc:302] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795e2135fc9546f98cfa6a259b717fae, a4d4990920624d97bcb8a5cb7a66373e; no voters:
I20260926 12:33:24.479522 12260 raft_consensus.cc:2781] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:33:24.479589 12260 raft_consensus.cc:482] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:33:24.479624 12260 raft_consensus.cc:3037] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:24.480937 12260 raft_consensus.cc:504] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } }
I20260926 12:33:24.481348 12260 leader_election.cc:288] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e [CANDIDATE]: Term 1 election: Requested vote from peers a774498f426c4efdadba4d25a124c4ae (127.10.98.1:36619), 795e2135fc9546f98cfa6a259b717fae (127.10.98.2:45441)
I20260926 12:33:24.481722 12045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f2a0e173a04636adba7fdf460b0073" candidate_uuid: "a4d4990920624d97bcb8a5cb7a66373e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "795e2135fc9546f98cfa6a259b717fae"
I20260926 12:33:24.481724 11905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b4f2a0e173a04636adba7fdf460b0073" candidate_uuid: "a4d4990920624d97bcb8a5cb7a66373e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a774498f426c4efdadba4d25a124c4ae"
I20260926 12:33:24.481839 11905 raft_consensus.cc:3037] T b4f2a0e173a04636adba7fdf460b0073 P a774498f426c4efdadba4d25a124c4ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:24.481844 12045 raft_consensus.cc:3037] T b4f2a0e173a04636adba7fdf460b0073 P 795e2135fc9546f98cfa6a259b717fae [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:24.482821 12045 raft_consensus.cc:2454] T b4f2a0e173a04636adba7fdf460b0073 P 795e2135fc9546f98cfa6a259b717fae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4d4990920624d97bcb8a5cb7a66373e in term 1.
I20260926 12:33:24.482934 11905 raft_consensus.cc:2454] T b4f2a0e173a04636adba7fdf460b0073 P a774498f426c4efdadba4d25a124c4ae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4d4990920624d97bcb8a5cb7a66373e in term 1.
I20260926 12:33:24.483088 12119 leader_election.cc:302] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 795e2135fc9546f98cfa6a259b717fae, a4d4990920624d97bcb8a5cb7a66373e; no voters:
I20260926 12:33:24.483297 12260 raft_consensus.cc:2781] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:24.483484 12260 raft_consensus.cc:686] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e [term 1 LEADER]: Becoming Leader. State: Replica: a4d4990920624d97bcb8a5cb7a66373e, State: Running, Role: LEADER
I20260926 12:33:24.483745 12260 consensus_queue.cc:237] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } }
I20260926 12:33:24.484927 12298 tablet_replica.cc:439] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e: TxnStatusTablet state changed. Reason: New leader a4d4990920624d97bcb8a5cb7a66373e. Latest consensus state: current_term: 1 leader_uuid: "a4d4990920624d97bcb8a5cb7a66373e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } } }
I20260926 12:33:24.485131 12298 tablet_replica.cc:442] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:33:24.485750 12300 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 12:33:24.485961 12300 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 12:33:24.486315 11747 catalog_manager.cc:5495] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e reported cstate change: term changed from 0 to 1, leader changed from <none> to a4d4990920624d97bcb8a5cb7a66373e (127.10.98.3). New cstate: current_term: 1 leader_uuid: "a4d4990920624d97bcb8a5cb7a66373e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } health_report { overall_health: HEALTHY } } }
I20260926 12:33:24.501632 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.497628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36018 (local address 127.10.98.3:43831)
0926 12:33:24.497734 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:24.497737 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:24.497834 (+ 97us) server_negotiation.cc:407] Connection header received
0926 12:33:24.498024 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:24.498026 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:24.498078 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:24.498173 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:24.498417 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.498934 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.500158 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.500384 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.500462 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.500764 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:24.501221 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:24.501307 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:24.501435 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 12:33:24.501485 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 12:33:24.501663 12292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.497537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:43831 (local address 127.0.0.1:36018)
0926 12:33:24.497768 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:24.497783 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:24.497893 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:24.498198 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:24.498200 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:24.498228 (+ 28us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:33:24.498334 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:24.498340 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.499036 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.499040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:24.500034 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:24.500045 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.500480 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.500484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:24.500574 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.500641 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:33:24.501313 (+ 672us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:33:24.501333 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 12:33:24.501417 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 12:33:24.501523 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260926 12:33:24.504594 12045 raft_consensus.cc:1261] T b4f2a0e173a04636adba7fdf460b0073 P 795e2135fc9546f98cfa6a259b717fae [term 1 FOLLOWER]: Refusing update from remote peer a4d4990920624d97bcb8a5cb7a66373e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:24.504703 11905 raft_consensus.cc:1261] T b4f2a0e173a04636adba7fdf460b0073 P a774498f426c4efdadba4d25a124c4ae [term 1 FOLLOWER]: Refusing update from remote peer a4d4990920624d97bcb8a5cb7a66373e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:24.505074 12298 consensus_queue.cc:1035] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e [LEADER]: Connected to new peer: Peer: permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:24.505265 12260 consensus_queue.cc:1035] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:24.506716 12262 tablet_replica.cc:439] T b4f2a0e173a04636adba7fdf460b0073 P 795e2135fc9546f98cfa6a259b717fae: TxnStatusTablet state changed. Reason: New leader a4d4990920624d97bcb8a5cb7a66373e. Latest consensus state: current_term: 1 leader_uuid: "a4d4990920624d97bcb8a5cb7a66373e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } } }
I20260926 12:33:24.506830 12262 tablet_replica.cc:442] T b4f2a0e173a04636adba7fdf460b0073 P 795e2135fc9546f98cfa6a259b717fae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:33:24.507179 12261 tablet_replica.cc:439] T b4f2a0e173a04636adba7fdf460b0073 P a774498f426c4efdadba4d25a124c4ae: TxnStatusTablet state changed. Reason: New leader a4d4990920624d97bcb8a5cb7a66373e. Latest consensus state: current_term: 1 leader_uuid: "a4d4990920624d97bcb8a5cb7a66373e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } } }
I20260926 12:33:24.507290 12261 tablet_replica.cc:442] T b4f2a0e173a04636adba7fdf460b0073 P a774498f426c4efdadba4d25a124c4ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:33:24.509037 12298 tablet_replica.cc:439] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4d4990920624d97bcb8a5cb7a66373e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } } }
I20260926 12:33:24.509183 12298 tablet_replica.cc:442] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:33:24.509287 12298 tablet_replica.cc:439] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a4d4990920624d97bcb8a5cb7a66373e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } } }
I20260926 12:33:24.509347 12298 tablet_replica.cc:442] T b4f2a0e173a04636adba7fdf460b0073 P a4d4990920624d97bcb8a5cb7a66373e: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:33:24.510843 12261 tablet_replica.cc:439] T b4f2a0e173a04636adba7fdf460b0073 P a774498f426c4efdadba4d25a124c4ae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4d4990920624d97bcb8a5cb7a66373e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } } }
I20260926 12:33:24.510958 12261 tablet_replica.cc:442] T b4f2a0e173a04636adba7fdf460b0073 P a774498f426c4efdadba4d25a124c4ae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:33:24.511793 12262 tablet_replica.cc:439] T b4f2a0e173a04636adba7fdf460b0073 P 795e2135fc9546f98cfa6a259b717fae: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a4d4990920624d97bcb8a5cb7a66373e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 } } peers { permanent_uuid: "795e2135fc9546f98cfa6a259b717fae" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 45441 } } peers { permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 } } }
I20260926 12:33:24.511907 12262 tablet_replica.cc:442] T b4f2a0e173a04636adba7fdf460b0073 P 795e2135fc9546f98cfa6a259b717fae: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:33:24.518299 12253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.514428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56622 (local address 127.10.98.2:45441)
0926 12:33:24.514534 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:24.514538 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:24.514552 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:33:24.514732 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:24.514735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:24.514793 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:24.514899 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:24.515085 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.515856 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.516586 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.516922 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.517020 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.517329 (+ 309us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:24.517905 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:24.518018 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:24.518110 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 12:33:24.518165 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 12:33:24.530210 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.525903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:36161 (local address 127.10.98.3:43831)
0926 12:33:24.526043 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:24.526047 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:24.526219 (+ 172us) server_negotiation.cc:407] Connection header received
0926 12:33:24.526395 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:24.526398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:24.526468 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:24.526553 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:24.526813 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.527381 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.529077 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.529347 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.529454 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.529788 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:24.529859 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:24.529910 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:24.530014 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 12:33:24.530061 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:33:24.530267 12307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.525778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:43831 (local address 127.10.98.2:36161)
0926 12:33:24.526139 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:24.526153 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:24.526257 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:24.526565 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:24.526568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:24.526576 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:33:24.526707 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:24.526713 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.527478 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.527482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:24.528902 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:24.528914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.529501 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.529506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:24.529609 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.529668 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:33:24.529933 (+ 265us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:33:24.529938 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 12:33:24.529998 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 12:33:24.530083 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":103,"threads_started":1}
I20260926 12:33:24.537487 12185 raft_consensus.cc:1261] T d39720085d8a411fb9e4858c539e5083 P a4d4990920624d97bcb8a5cb7a66373e [term 1 FOLLOWER]: Refusing update from remote peer 795e2135fc9546f98cfa6a259b717fae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:24.537531 11905 raft_consensus.cc:1261] T d39720085d8a411fb9e4858c539e5083 P a774498f426c4efdadba4d25a124c4ae [term 1 FOLLOWER]: Refusing update from remote peer 795e2135fc9546f98cfa6a259b717fae: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:24.537987 12285 consensus_queue.cc:1035] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4d4990920624d97bcb8a5cb7a66373e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 43831 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:24.538275 12262 consensus_queue.cc:1035] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae [LEADER]: Connected to new peer: Peer: permanent_uuid: "a774498f426c4efdadba4d25a124c4ae" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:24.562489 12253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.558375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:46079 (local address 127.10.98.2:45441)
0926 12:33:24.558500 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:24.558505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:24.558629 (+ 124us) server_negotiation.cc:407] Connection header received
0926 12:33:24.558728 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:24.558731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:24.558781 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:24.558861 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:24.559158 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.559955 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.561318 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.561566 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.561650 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.561971 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:24.562068 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:24.562135 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:24.562266 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 12:33:24.562329 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:33:24.562927 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.558140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:45441 (local address 127.10.98.3:46079)
0926 12:33:24.558550 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:24.558564 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:24.558656 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:24.558865 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:24.558866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:24.558872 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:33:24.559020 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:24.559027 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.560110 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.560113 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:24.561200 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:24.561213 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.561687 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.561690 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:24.561786 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.561845 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:33:24.562150 (+ 305us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:33:24.562157 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 12:33:24.562695 (+ 538us) client_negotiation.cc:240] Negotiation successful
0926 12:33:24.562783 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":178,"threads_started":1}
I20260926 12:33:24.592273 11748 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:24.592464 11748 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:24.595027 11748 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c7cd35a462414c5abe243c2c5de50ad4: Sending DeleteTablet for 3 replicas of tablet d39720085d8a411fb9e4858c539e5083
I20260926 12:33:24.595750 12025 tablet_service.cc:1514] Processing DeleteTablet for tablet d39720085d8a411fb9e4858c539e5083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:24 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:56606
I20260926 12:33:24.595922 11885 tablet_service.cc:1514] Processing DeleteTablet for tablet d39720085d8a411fb9e4858c539e5083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:24 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:32872
I20260926 12:33:24.596124 12279 tablet_replica.cc:330] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae: stopping tablet replica
I20260926 12:33:24.596249 12279 raft_consensus.cc:2229] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:33:24.596561 12279 raft_consensus.cc:2258] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:24.596774 12277 tablet_replica.cc:330] T d39720085d8a411fb9e4858c539e5083 P a774498f426c4efdadba4d25a124c4ae: stopping tablet replica
I20260926 12:33:24.596895 12277 raft_consensus.cc:2229] T d39720085d8a411fb9e4858c539e5083 P a774498f426c4efdadba4d25a124c4ae [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:24.597003 12277 raft_consensus.cc:2258] T d39720085d8a411fb9e4858c539e5083 P a774498f426c4efdadba4d25a124c4ae [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:24.597462 12279 ts_tablet_manager.cc:1905] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:24.597983 12277 ts_tablet_manager.cc:1905] T d39720085d8a411fb9e4858c539e5083 P a774498f426c4efdadba4d25a124c4ae: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:24.600373 12279 ts_tablet_manager.cc:1918] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:33:24.600448 12279 log.cc:1196] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/d39720085d8a411fb9e4858c539e5083
I20260926 12:33:24.600700 12279 ts_tablet_manager.cc:1939] T d39720085d8a411fb9e4858c539e5083 P 795e2135fc9546f98cfa6a259b717fae: Deleting consensus metadata
I20260926 12:33:24.600723 12277 ts_tablet_manager.cc:1918] T d39720085d8a411fb9e4858c539e5083 P a774498f426c4efdadba4d25a124c4ae: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:33:24.600791 12277 log.cc:1196] T d39720085d8a411fb9e4858c539e5083 P a774498f426c4efdadba4d25a124c4ae: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-0/wal/wals/d39720085d8a411fb9e4858c539e5083
I20260926 12:33:24.601011 12277 ts_tablet_manager.cc:1939] T d39720085d8a411fb9e4858c539e5083 P a774498f426c4efdadba4d25a124c4ae: Deleting consensus metadata
I20260926 12:33:24.601717 11735 catalog_manager.cc:4841] TS 795e2135fc9546f98cfa6a259b717fae (127.10.98.2:45441): tablet d39720085d8a411fb9e4858c539e5083 (table test-table [id=17b4346399f743e19a634957b2605e3e]) successfully deleted
I20260926 12:33:24.601950 12165 tablet_service.cc:1514] Processing DeleteTablet for tablet d39720085d8a411fb9e4858c539e5083 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:24 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:36014
I20260926 12:33:24.602123 12278 tablet_replica.cc:330] T d39720085d8a411fb9e4858c539e5083 P a4d4990920624d97bcb8a5cb7a66373e: stopping tablet replica
I20260926 12:33:24.602221 12278 raft_consensus.cc:2229] T d39720085d8a411fb9e4858c539e5083 P a4d4990920624d97bcb8a5cb7a66373e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:24.602324 12278 raft_consensus.cc:2258] T d39720085d8a411fb9e4858c539e5083 P a4d4990920624d97bcb8a5cb7a66373e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:24.603014 11733 catalog_manager.cc:4841] TS a774498f426c4efdadba4d25a124c4ae (127.10.98.1:36619): tablet d39720085d8a411fb9e4858c539e5083 (table test-table [id=17b4346399f743e19a634957b2605e3e]) successfully deleted
I20260926 12:33:24.603044 12278 ts_tablet_manager.cc:1905] T d39720085d8a411fb9e4858c539e5083 P a4d4990920624d97bcb8a5cb7a66373e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:24.605654 12278 ts_tablet_manager.cc:1918] T d39720085d8a411fb9e4858c539e5083 P a4d4990920624d97bcb8a5cb7a66373e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:33:24.605726 12278 log.cc:1196] T d39720085d8a411fb9e4858c539e5083 P a4d4990920624d97bcb8a5cb7a66373e: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/d39720085d8a411fb9e4858c539e5083
I20260926 12:33:24.605933 12278 ts_tablet_manager.cc:1939] T d39720085d8a411fb9e4858c539e5083 P a4d4990920624d97bcb8a5cb7a66373e: Deleting consensus metadata
I20260926 12:33:24.606739 11735 catalog_manager.cc:4841] TS a4d4990920624d97bcb8a5cb7a66373e (127.10.98.3:43831): tablet d39720085d8a411fb9e4858c539e5083 (table test-table [id=17b4346399f743e19a634957b2605e3e]) successfully deleted
Sep 26 12:33:24 dist-test-slave-t7tk krb5kdc[11703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426003, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.98.1@KRBTEST.COM
I20260926 12:33:24.615549 12252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.603576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32886 (local address 127.10.98.1:36619)
0926 12:33:24.603691 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:24.603696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:24.603961 (+ 265us) server_negotiation.cc:407] Connection header received
0926 12:33:24.604104 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:24.604107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:24.604155 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:24.604231 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:24.605039 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.605863 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.607178 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.607520 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.607624 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.611243 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:24.611257 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:24.611273 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:24.611302 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:24.613339 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:24.613847 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:24.613854 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:24.613856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:24.613949 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:24.614241 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:24.614244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:24.614246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:24.614585 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:24.614885 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:24.615090 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 12:33:24.615220 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 12:33:24.616317 11945 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32886
I20260926 12:33:24.627795 11797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:24.618536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35502 (local address 127.10.98.62:43177)
0926 12:33:24.618662 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:24.618667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:24.619102 (+ 435us) server_negotiation.cc:407] Connection header received
0926 12:33:24.619222 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:24.619225 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:24.619280 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:24.619363 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:24.620124 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.620979 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.622203 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:24.622556 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:24.622677 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:24.624288 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:24.624293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:24.624296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:24.624324 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:24.625990 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:24.626492 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:24.626495 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:24.626497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:24.626597 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:24.626878 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:24.626881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:24.626883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:24.627102 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:24.627228 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:24.627497 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 12:33:24.627629 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 12:33:24.628469 11748 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35502
I20260926 12:33:24.629910 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 11820
I20260926 12:33:24.639467 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 11960
I20260926 12:33:24.645272 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 12100
I20260926 12:33:24.650893 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 11716
2026-09-26T12:33:24Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2746 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:33:24 dist-test-slave-t7tk krb5kdc[12322](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:33:24 dist-test-slave-t7tk krb5kdc[12322](info): set up 2 sockets
Sep 26 12:33:24 dist-test-slave-t7tk krb5kdc[12322](info): commencing operation
krb5kdc: starting...
W20260926 12:33:25.614344 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:33:25 dist-test-slave-t7tk krb5kdc[12322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:33:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:33:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:33:25.736670 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:41413
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:38151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:41413
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:25.845945 12335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:25.846230 12335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:25.846269 12335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:25.850214 12335 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:25.850271 12335 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 12:33:25.850286 12335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:25.850302 12335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:25.850317 12335 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:25.850330 12335 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:33:25.856021 12335 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:38151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:41413
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:41413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:25.857059 12335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:25.858054 12335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:33:25.866922 12335 server_base.cc:1034] running on GCE node
W20260926 12:33:25.866856 12341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:25.866856 12343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:25.866915 12340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:25.867755 12335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:25.868955 12335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:25.870137 12335 hybrid_clock.cc:648] HybridClock initialized: now 1790426005870107 us; error 44 us; skew 500 ppm
Sep 26 12:33:25 dist-test-slave-t7tk krb5kdc[12322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426005, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:25.879019 12335 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:33:25.880004 12335 webserver.cc:460] Webserver started at http://127.10.98.62:42893/ using document root <none> and password file <none>
I20260926 12:33:25.880537 12335 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:25.880596 12335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:25.880782 12335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:25.882977 12335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "6330d03515904304a66a0196d784af0c"
format_stamp: "Formatted at 2026-09-26 12:33:25 on dist-test-slave-t7tk"
server_key: "368c49ce2c715a04f74dd1c20634ba7d"
server_key_iv: "b52eabecae9c45a04298fb5cd85ecbfc"
server_key_version: "encryptionkey@0"
I20260926 12:33:25.883477 12335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "6330d03515904304a66a0196d784af0c"
format_stamp: "Formatted at 2026-09-26 12:33:25 on dist-test-slave-t7tk"
server_key: "368c49ce2c715a04f74dd1c20634ba7d"
server_key_iv: "b52eabecae9c45a04298fb5cd85ecbfc"
server_key_version: "encryptionkey@0"
I20260926 12:33:25.887217 12335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:33:25.889832 12350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:25.891310 12335 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:33:25.891427 12335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "6330d03515904304a66a0196d784af0c"
format_stamp: "Formatted at 2026-09-26 12:33:25 on dist-test-slave-t7tk"
server_key: "368c49ce2c715a04f74dd1c20634ba7d"
server_key_iv: "b52eabecae9c45a04298fb5cd85ecbfc"
server_key_version: "encryptionkey@0"
I20260926 12:33:25.891523 12335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:25.911763 12335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:25.915200 12335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:25.915376 12335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:25.923240 12335 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:41413
I20260926 12:33:25.923281 12412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:41413 every 8 connection(s)
I20260926 12:33:25.924193 12335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:33:25.927122 12413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:25.932963 12413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c: Bootstrap starting.
I20260926 12:33:25.933463 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 12335
I20260926 12:33:25.933580 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:33:25.933836 10632 external_mini_cluster.cc:1428] Setting key 1ca663e4065b702edd67fbe82c1e9057
I20260926 12:33:25.935883 12413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:25.936606 12413 log.cc:824] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:25.938805 12413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c: No bootstrap required, opened a new log
I20260926 12:33:25.943019 12413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6330d03515904304a66a0196d784af0c" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41413 } }
Sep 26 12:33:25 dist-test-slave-t7tk krb5kdc[12322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426005, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:25.943269 12413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:25.943308 12413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6330d03515904304a66a0196d784af0c, State: Initialized, Role: FOLLOWER
I20260926 12:33:25.943831 12413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6330d03515904304a66a0196d784af0c" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41413 } }
I20260926 12:33:25.943982 12413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:33:25.944051 12413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:33:25.944147 12413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:25.945209 12413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6330d03515904304a66a0196d784af0c" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41413 } }
I20260926 12:33:25.945552 12413 leader_election.cc:302] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6330d03515904304a66a0196d784af0c; no voters:
I20260926 12:33:25.945852 12413 leader_election.cc:288] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:33:25.945955 12418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:25.946169 12418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c [term 1 LEADER]: Becoming Leader. State: Replica: 6330d03515904304a66a0196d784af0c, State: Running, Role: LEADER
I20260926 12:33:25.946480 12418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6330d03515904304a66a0196d784af0c" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41413 } }
I20260926 12:33:25.947211 12413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:25.948069 12420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6330d03515904304a66a0196d784af0c. Latest consensus state: current_term: 1 leader_uuid: "6330d03515904304a66a0196d784af0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6330d03515904304a66a0196d784af0c" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41413 } } }
I20260926 12:33:25.948197 12420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:25.949193 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:25.935346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45824 (local address 127.10.98.62:41413)
0926 12:33:25.935806 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:25.935821 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:25.935870 (+ 49us) server_negotiation.cc:407] Connection header received
0926 12:33:25.936571 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:25.936590 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:25.936970 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:25.937463 (+ 493us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:25.938278 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:25.939308 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:25.940469 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:25.940997 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:25.941149 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:25.943872 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:25.943876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:25.943890 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:25.943919 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:25.945769 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:25.946269 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:25.946275 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:25.946282 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:25.946417 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:25.946810 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:25.946813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:25.946815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:25.947343 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:25.947538 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:25.947846 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 12:33:25.948167 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":135,"threads_started":1}
I20260926 12:33:25.949680 12419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6330d03515904304a66a0196d784af0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6330d03515904304a66a0196d784af0c" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41413 } } }
I20260926 12:33:25.950014 12419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:25.950100 12427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:33:25.952593 12427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:33:25.958139 12427 catalog_manager.cc:1347] Generated new cluster ID: 270ab5f0a5ee46b5a0735b4c71463c7f
I20260926 12:33:25.958207 12427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:33:25.991497 12427 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:33:25.992544 12427 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:33:26.001770 12427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c: Generated new TSK 0
I20260926 12:33:26.002547 12427 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
I20260926 12:33:26.037279 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:41413
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:38151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:26.173717 12439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:26.173986 12439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:26.174033 12439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:26.178092 12439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:26.178160 12439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:26.178200 12439 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 12:33:26.178213 12439 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:33:26.178244 12439 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:33:26.183655 12439 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:38151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.98.62:41413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12439
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:26.184697 12439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:26.185702 12439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:26.195010 12444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:26.195063 12447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:26.195026 12445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:26.195376 12439 server_base.cc:1034] running on GCE node
I20260926 12:33:26.195817 12439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:26.196518 12439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:26.197677 12439 hybrid_clock.cc:648] HybridClock initialized: now 1790426006197648 us; error 46 us; skew 500 ppm
Sep 26 12:33:26 dist-test-slave-t7tk krb5kdc[12322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426006, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:26.200278 12439 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:33:26.201224 12439 webserver.cc:460] Webserver started at http://127.10.98.1:45575/ using document root <none> and password file <none>
I20260926 12:33:26.201768 12439 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:26.201831 12439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:26.202019 12439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:26.204723 12439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "e485ea0dfa084e81b7c8205293de6527"
format_stamp: "Formatted at 2026-09-26 12:33:26 on dist-test-slave-t7tk"
server_key: "38d5b3953ce7b2969a6bffc35c7be83e"
server_key_iv: "4c9e27eef7d39a766eff36627624b00b"
server_key_version: "encryptionkey@0"
I20260926 12:33:26.205421 12439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "e485ea0dfa084e81b7c8205293de6527"
format_stamp: "Formatted at 2026-09-26 12:33:26 on dist-test-slave-t7tk"
server_key: "38d5b3953ce7b2969a6bffc35c7be83e"
server_key_iv: "4c9e27eef7d39a766eff36627624b00b"
server_key_version: "encryptionkey@0"
I20260926 12:33:26.209120 12439 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260926 12:33:26.211585 12454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:26.212881 12439 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:33:26.213021 12439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "e485ea0dfa084e81b7c8205293de6527"
format_stamp: "Formatted at 2026-09-26 12:33:26 on dist-test-slave-t7tk"
server_key: "38d5b3953ce7b2969a6bffc35c7be83e"
server_key_iv: "4c9e27eef7d39a766eff36627624b00b"
server_key_version: "encryptionkey@0"
I20260926 12:33:26.213188 12439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:26.233080 12439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:26.236246 12439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:26.236423 12439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:26.237919 12439 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:26.237970 12439 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:26.238024 12439 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:26.238051 12439 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:33:26 dist-test-slave-t7tk krb5kdc[12322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426006, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:26.247727 12439 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:41323
I20260926 12:33:26.247782 12569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:41323 every 8 connection(s)
I20260926 12:33:26.248720 12439 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:33:26.251147 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.238612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:34659 (local address 127.10.98.62:41413)
0926 12:33:26.238760 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:26.238765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:26.239639 (+ 874us) server_negotiation.cc:407] Connection header received
0926 12:33:26.240758 (+ 1119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:26.240763 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:26.240829 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:26.240891 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:26.242343 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.242861 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.244033 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.244398 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.244498 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.247484 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:26.247488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:26.247491 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:26.247522 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:26.249026 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:26.249616 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:26.249620 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:26.249621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:26.249719 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:26.250110 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:26.250113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:26.250114 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:26.250313 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:26.250396 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:26.250878 (+ 482us) server_negotiation.cc:299] Negotiation successful
0926 12:33:26.251004 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:33:26.251853 12463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.239016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:41413 (local address 127.10.98.1:34659)
0926 12:33:26.239472 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:26.239530 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:26.240491 (+ 961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:26.241330 (+ 839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:26.241339 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:26.241686 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:26.242198 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.242212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.242978 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.242982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.243898 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.243908 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.244566 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.244571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.244774 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.245891 (+ 1117us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:26.245907 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:26.247298 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:26.249152 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:26.249158 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:26.249173 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:26.249498 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:26.249830 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:26.249833 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:26.249834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:26.249963 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:26.250401 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:26.250406 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:26.250646 (+ 240us) client_negotiation.cc:769] Sending connection context
0926 12:33:26.250793 (+ 147us) client_negotiation.cc:240] Negotiation successful
0926 12:33:26.251043 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":110,"threads_started":1}
I20260926 12:33:26.253206 12570 heartbeater.cc:347] Connected to a master server at 127.10.98.62:41413
I20260926 12:33:26.253355 12570 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:26.253856 12570 heartbeater.cc:511] Master 127.10.98.62:41413 requested a full tablet report, sending...
I20260926 12:33:26.254559 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 12439
I20260926 12:33:26.254674 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:33:26.254966 10632 external_mini_cluster.cc:1428] Setting key 12ff99bf16cd98bcb041d5e97651c214
I20260926 12:33:26.255571 12366 ts_manager.cc:194] Registered new tserver with Master: e485ea0dfa084e81b7c8205293de6527 (127.10.98.1:41323)
I20260926 12:33:26.256664 12366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:34659
I20260926 12:33:26.265602 12576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.255325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:41413 (local address 127.10.98.1:59401)
0926 12:33:26.255574 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:26.255587 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:26.255660 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:26.259100 (+ 3440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:26.259104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:26.259316 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:26.259431 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.259438 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.260230 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.260234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.261082 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.261125 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.261570 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.261574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.261664 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.262170 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:26.262184 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:26.262635 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:26.264210 (+ 1575us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:26.264213 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:26.264215 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:26.264405 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:26.264646 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:26.264649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:26.264651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:26.264728 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:26.265062 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:26.265065 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:26.265178 (+ 113us) client_negotiation.cc:769] Sending connection context
0926 12:33:26.265266 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 12:33:26.265407 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":83,"threads_started":1}
I20260926 12:33:26.265596 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.257246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:59401 (local address 127.10.98.62:41413)
0926 12:33:26.257354 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:26.257357 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:26.257371 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:33:26.257403 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:26.257405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:26.257442 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:26.257512 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:26.259570 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.260104 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.261225 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.261456 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.261543 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.262763 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:26.262766 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:26.262768 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:26.262792 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:26.264103 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:26.264492 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:26.264496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:26.264498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:26.264567 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:26.264805 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:26.264808 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:26.264810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:26.264977 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:26.265058 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:26.265326 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 12:33:26.265428 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":37}
WARNING: no policy specified for kudu/127.10.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
I20260926 12:33:26.284273 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:41413
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:38151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:26.393818 12579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:26.394081 12579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:26.394130 12579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:26.398201 12579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:26.398258 12579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:26.398298 12579 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 12:33:26.398311 12579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:33:26.398363 12579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:33:26.403566 12579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:38151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.98.62:41413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:26.404589 12579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:26.405674 12579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:26.415370 12587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:26.415397 12585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:26.415426 12584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:26.415618 12579 server_base.cc:1034] running on GCE node
I20260926 12:33:26.416059 12579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:26.416704 12579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:26.417865 12579 hybrid_clock.cc:648] HybridClock initialized: now 1790426006417788 us; error 108 us; skew 500 ppm
Sep 26 12:33:26 dist-test-slave-t7tk krb5kdc[12322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426006, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:26.420678 12579 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:33:26.421811 12579 webserver.cc:460] Webserver started at http://127.10.98.2:37995/ using document root <none> and password file <none>
I20260926 12:33:26.422374 12579 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:26.422454 12579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:26.422676 12579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:26.424736 12579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "ecb0141621ca4a1c9d4f8e7f65f76653"
format_stamp: "Formatted at 2026-09-26 12:33:26 on dist-test-slave-t7tk"
server_key: "29e485028c907125c7017e25b259d276"
server_key_iv: "8f2402561ba651dae3078a9fc274f042"
server_key_version: "encryptionkey@0"
I20260926 12:33:26.425309 12579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "ecb0141621ca4a1c9d4f8e7f65f76653"
format_stamp: "Formatted at 2026-09-26 12:33:26 on dist-test-slave-t7tk"
server_key: "29e485028c907125c7017e25b259d276"
server_key_iv: "8f2402561ba651dae3078a9fc274f042"
server_key_version: "encryptionkey@0"
I20260926 12:33:26.429028 12579 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:33:26.431509 12594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:26.432819 12579 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 12:33:26.432934 12579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "ecb0141621ca4a1c9d4f8e7f65f76653"
format_stamp: "Formatted at 2026-09-26 12:33:26 on dist-test-slave-t7tk"
server_key: "29e485028c907125c7017e25b259d276"
server_key_iv: "8f2402561ba651dae3078a9fc274f042"
server_key_version: "encryptionkey@0"
I20260926 12:33:26.433034 12579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:26.449813 12579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:26.452863 12579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:26.453037 12579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:26.454535 12579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:26.454586 12579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:26.454627 12579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:26.454655 12579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:33:26 dist-test-slave-t7tk krb5kdc[12322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426006, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:26.465262 12579 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:42427
I20260926 12:33:26.465342 12709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:42427 every 8 connection(s)
I20260926 12:33:26.466233 12579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:33:26.468205 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.455372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:48341 (local address 127.10.98.62:41413)
0926 12:33:26.455484 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:26.455488 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:26.456561 (+ 1073us) server_negotiation.cc:407] Connection header received
0926 12:33:26.457615 (+ 1054us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:26.457618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:26.457675 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:26.457753 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:26.459008 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.459551 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.460851 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.461160 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.461262 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.464509 (+ 3247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:26.464512 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:26.464514 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:26.464541 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:26.466075 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:26.466657 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:26.466661 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:26.466663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:26.466761 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:26.467143 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:26.467147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:26.467149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:26.467363 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:26.467445 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:26.467917 (+ 472us) server_negotiation.cc:299] Negotiation successful
0926 12:33:26.468077 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 12:33:26.468924 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.455910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:41413 (local address 127.10.98.2:48341)
0926 12:33:26.456405 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:26.456458 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:26.457400 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:26.457941 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:26.457948 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:26.458345 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:26.458867 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.458883 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.459701 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.459705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.460726 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.460736 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.461377 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.461382 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.461599 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.462327 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:26.462347 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:26.464332 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:26.466197 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:26.466203 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:26.466218 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:26.466555 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:26.466874 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:26.466877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:26.466879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:26.467041 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:26.467441 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:26.467446 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:26.467670 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 12:33:26.467825 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 12:33:26.468085 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":132,"threads_started":1}
I20260926 12:33:26.470176 12710 heartbeater.cc:347] Connected to a master server at 127.10.98.62:41413
I20260926 12:33:26.470316 12710 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:26.470407 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 12579
I20260926 12:33:26.470546 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:33:26.470821 10632 external_mini_cluster.cc:1428] Setting key 03ceaf28a6ba5b0fed2b540f9873f85c
I20260926 12:33:26.470825 12710 heartbeater.cc:511] Master 127.10.98.62:41413 requested a full tablet report, sending...
I20260926 12:33:26.471844 12366 ts_manager.cc:194] Registered new tserver with Master: ecb0141621ca4a1c9d4f8e7f65f76653 (127.10.98.2:42427)
I20260926 12:33:26.472537 12366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:48341
WARNING: no policy specified for kudu/127.10.98.3@KRBTEST.COM; defaulting to no policy
I20260926 12:33:26.480450 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.472664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:60209 (local address 127.10.98.62:41413)
0926 12:33:26.472781 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:26.472785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:26.473222 (+ 437us) server_negotiation.cc:407] Connection header received
0926 12:33:26.473387 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:26.473390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:26.473432 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:26.473482 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:26.474109 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.474850 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.475630 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.475952 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.476056 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.477062 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:26.477067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:26.477069 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:26.477127 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:26.478554 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:26.479006 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:26.479010 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:26.479012 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:26.479109 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:26.479362 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:26.479365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:26.479367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:26.479660 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:26.479759 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:26.480171 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 12:33:26.480281 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:33:26.480455 12716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.472848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:41413 (local address 127.10.98.2:60209)
0926 12:33:26.473142 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:26.473157 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:26.473288 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:26.473611 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:26.473614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:26.473820 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:26.473967 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.473976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.474977 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.474979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.475528 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.475536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.476062 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.476065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.476132 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.476579 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:26.476590 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:26.476899 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:26.478670 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:26.478673 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:26.478674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:26.478912 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:26.479200 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:26.479203 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:26.479204 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:26.479277 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:26.479775 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:26.479777 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:26.479868 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 12:33:26.480070 (+ 202us) client_negotiation.cc:240] Negotiation successful
0926 12:33:26.480226 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":89,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
I20260926 12:33:26.500314 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:41413
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:38151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:26.607316 12719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:26.607579 12719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:26.607625 12719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:26.611680 12719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:26.611742 12719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:26.611795 12719 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 12:33:26.611825 12719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:33:26.611878 12719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:33:26.617451 12719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:38151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.98.62:41413
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:26.618479 12719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:26.619490 12719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:26.628878 12727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:26.628882 12725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:26.629014 12724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:26.629004 12719 server_base.cc:1034] running on GCE node
I20260926 12:33:26.629591 12719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:26.630203 12719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:26.631376 12719 hybrid_clock.cc:648] HybridClock initialized: now 1790426006631345 us; error 44 us; skew 500 ppm
Sep 26 12:33:26 dist-test-slave-t7tk krb5kdc[12322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426006, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:26.634030 12719 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:33:26.634940 12719 webserver.cc:460] Webserver started at http://127.10.98.3:46699/ using document root <none> and password file <none>
I20260926 12:33:26.635541 12719 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:26.635617 12719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:26.635808 12719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:26.637912 12719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "ab7caa772a3844ea94caf798f45a9a4a"
format_stamp: "Formatted at 2026-09-26 12:33:26 on dist-test-slave-t7tk"
server_key: "0bcdc13cb29e8599b09375f1df428bd8"
server_key_iv: "4b90f0fe352907d9615dab2177cf1141"
server_key_version: "encryptionkey@0"
I20260926 12:33:26.638441 12719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "ab7caa772a3844ea94caf798f45a9a4a"
format_stamp: "Formatted at 2026-09-26 12:33:26 on dist-test-slave-t7tk"
server_key: "0bcdc13cb29e8599b09375f1df428bd8"
server_key_iv: "4b90f0fe352907d9615dab2177cf1141"
server_key_version: "encryptionkey@0"
I20260926 12:33:26.642092 12719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:33:26.644538 12734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:26.645839 12719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:33:26.645943 12719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "ab7caa772a3844ea94caf798f45a9a4a"
format_stamp: "Formatted at 2026-09-26 12:33:26 on dist-test-slave-t7tk"
server_key: "0bcdc13cb29e8599b09375f1df428bd8"
server_key_iv: "4b90f0fe352907d9615dab2177cf1141"
server_key_version: "encryptionkey@0"
I20260926 12:33:26.646039 12719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:26.668290 12719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:26.671854 12719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:26.672034 12719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:26.673519 12719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:26.673568 12719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:26.673611 12719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:26.673640 12719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:26.683240 12719 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:33653
I20260926 12:33:26.683283 12849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:33653 every 8 connection(s)
Sep 26 12:33:26 dist-test-slave-t7tk krb5kdc[12322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426006, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:26.684247 12719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:33:26.686476 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 12719
I20260926 12:33:26.686578 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:33:26.686882 10632 external_mini_cluster.cc:1428] Setting key 21e7eb1698b4afb39ab95fdbf568a1f2
I20260926 12:33:26.687855 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.674420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:37297 (local address 127.10.98.62:41413)
0926 12:33:26.674557 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:26.674560 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:26.675633 (+ 1073us) server_negotiation.cc:407] Connection header received
0926 12:33:26.676817 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:26.676820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:26.676880 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:26.676954 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:26.678461 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.679252 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.680843 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.681255 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.681395 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.684118 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:26.684121 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:26.684123 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:26.684149 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:26.685580 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:26.686038 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:26.686042 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:26.686044 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:26.686243 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:26.686633 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:26.686636 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:26.686637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:26.686873 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:26.686973 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:26.687540 (+ 567us) server_negotiation.cc:299] Negotiation successful
0926 12:33:26.687700 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 12:33:26.688664 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.674980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:41413 (local address 127.10.98.3:37297)
0926 12:33:26.675476 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:26.675525 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:26.676553 (+ 1028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:26.677263 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:26.677272 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:26.677651 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:26.678323 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.678342 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.679372 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.679375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.680720 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.680729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.681410 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.681415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.681611 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.682268 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:26.682289 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:26.683912 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:26.685661 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:26.685665 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:26.685676 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:26.685929 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:26.686366 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:26.686369 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:26.686370 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:26.686514 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:26.687028 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:26.687034 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:26.687290 (+ 256us) client_negotiation.cc:769] Sending connection context
0926 12:33:26.687477 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 12:33:26.687753 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":134,"threads_started":1}
I20260926 12:33:26.689874 12850 heartbeater.cc:347] Connected to a master server at 127.10.98.62:41413
I20260926 12:33:26.690025 12850 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:26.690507 12850 heartbeater.cc:511] Master 127.10.98.62:41413 requested a full tablet report, sending...
I20260926 12:33:26.691439 12366 ts_manager.cc:194] Registered new tserver with Master: ab7caa772a3844ea94caf798f45a9a4a (127.10.98.3:33653)
I20260926 12:33:26.692003 12366 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:37297
I20260926 12:33:26.698504 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.691870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:50231 (local address 127.10.98.62:41413)
0926 12:33:26.691981 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:26.691985 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:26.692145 (+ 160us) server_negotiation.cc:407] Connection header received
0926 12:33:26.692260 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:26.692263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:26.692311 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:26.692383 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:26.692864 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.693678 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.694529 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.694747 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.694824 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.695843 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:26.695846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:26.695848 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:26.695870 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:26.697067 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:26.697487 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:26.697489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:26.697490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:26.697560 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:26.697839 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:26.697841 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:26.697842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:26.697994 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:26.698063 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:26.698302 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 12:33:26.698391 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:33:26.698518 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.691801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:41413 (local address 127.10.98.3:50231)
0926 12:33:26.692073 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:26.692087 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:26.692181 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:26.692410 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:26.692413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:26.692623 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:26.692758 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.692763 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.693810 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.693813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.694416 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.694425 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.694862 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.694864 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.694955 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.695414 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:26.695424 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:26.695723 (+ 299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:26.697186 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:26.697188 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:26.697190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:26.697389 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:26.697676 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:26.697679 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:26.697680 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:26.697765 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:26.698092 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:26.698094 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:26.698177 (+ 83us) client_negotiation.cc:769] Sending connection context
0926 12:33:26.698263 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 12:33:26.698383 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260926 12:33:26.700471 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 12:33:26 dist-test-slave-t7tk krb5kdc[12322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 12:33:26 dist-test-slave-t7tk krb5kdc[12322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:26.724555 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.716492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45836 (local address 127.10.98.62:41413)
0926 12:33:26.716672 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:26.716676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:26.716689 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:33:26.716729 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:26.716732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:26.716772 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:26.716844 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:26.717574 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.718142 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.718993 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.719222 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.719304 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.721670 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:26.721673 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:26.721675 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:26.721703 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:26.723153 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:26.723499 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:26.723501 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:26.723502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:26.723571 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:26.723757 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:26.723758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:26.723760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:26.723925 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:26.724068 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:26.724290 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 12:33:26.724385 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:33:26.727702 12366 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:33:26.730715 12366 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:33:26.745126 12867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.739456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:42427 (local address 127.0.0.1:32826)
0926 12:33:26.740223 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:26.740260 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:26.740393 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:26.740955 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:26.740961 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:26.740985 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:26.741157 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.741166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.742238 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.742241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.743499 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.743509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.744568 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.744571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.744677 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.744718 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 12:33:26.744823 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 12:33:26.744946 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260926 12:33:26.745133 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.739572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32826 (local address 127.10.98.2:42427)
0926 12:33:26.740101 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:26.740114 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:26.740344 (+ 230us) server_negotiation.cc:407] Connection header received
0926 12:33:26.740540 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:26.740545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:26.740802 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:26.740968 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:26.741259 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.742109 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.743637 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.744461 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.744573 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.744759 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:26.744851 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 12:33:26.744947 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":134,"threads_started":1}
I20260926 12:33:26.745920 12869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.739805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:41323 (local address 127.0.0.1:58766)
0926 12:33:26.741064 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:26.741075 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:26.741172 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:26.741611 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:26.741614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:26.741630 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:26.741730 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.741737 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.743075 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.743078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.744315 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.744336 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.745530 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.745533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.745626 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.745650 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:33:26.745715 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 12:33:26.745781 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1209,"mutex_wait_us":40,"thread_start_us":73,"threads_started":1}
I20260926 12:33:26.745925 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.739576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:33653 (local address 127.0.0.1:59848)
0926 12:33:26.740822 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:26.740834 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:26.740915 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:26.742211 (+ 1296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:26.742213 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:26.742229 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:26.742330 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.742336 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.743316 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.743319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.744490 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.744500 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.745606 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.745610 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.745690 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.745709 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 12:33:26.745758 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 12:33:26.745806 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1186,"mutex_wait_us":2,"thread_start_us":125,"threads_started":1}
I20260926 12:33:26.746017 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.740243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59848 (local address 127.10.98.3:33653)
0926 12:33:26.741890 (+ 1647us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:26.741899 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:26.741917 (+ 18us) server_negotiation.cc:407] Connection header received
0926 12:33:26.741970 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:26.741974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:26.742098 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:26.742215 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:26.742427 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.743207 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.744671 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.745500 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.745608 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.745759 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:26.745802 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 12:33:26.745884 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1547,"thread_start_us":97,"threads_started":1}
I20260926 12:33:26.746361 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.740245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.10.98.1:41323)
0926 12:33:26.740687 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:26.740701 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:26.741257 (+ 556us) server_negotiation.cc:407] Connection header received
0926 12:33:26.741329 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:26.741334 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:26.741482 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:26.741614 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:26.741871 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.742962 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.744436 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.745214 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.745310 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.745477 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:26.746140 (+ 663us) server_negotiation.cc:299] Negotiation successful
0926 12:33:26.746223 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":81,"threads_started":1}
I20260926 12:33:26.748719 12504 tablet_service.cc:1467] Processing CreateTablet for tablet c8f90aa888a6494ea718cc52ed399502 (DEFAULT_TABLE table=test-table [id=f84bcf98e4964a4c9fb5ffe5b5356d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:26.748628 12644 tablet_service.cc:1467] Processing CreateTablet for tablet c8f90aa888a6494ea718cc52ed399502 (DEFAULT_TABLE table=test-table [id=f84bcf98e4964a4c9fb5ffe5b5356d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:26.748628 12784 tablet_service.cc:1467] Processing CreateTablet for tablet c8f90aa888a6494ea718cc52ed399502 (DEFAULT_TABLE table=test-table [id=f84bcf98e4964a4c9fb5ffe5b5356d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:26.749675 12784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8f90aa888a6494ea718cc52ed399502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:26.749670 12644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8f90aa888a6494ea718cc52ed399502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:26.749677 12504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8f90aa888a6494ea718cc52ed399502. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:26.756826 12873 tablet_bootstrap.cc:492] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527: Bootstrap starting.
I20260926 12:33:26.757541 12874 tablet_bootstrap.cc:492] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a: Bootstrap starting.
I20260926 12:33:26.759076 12873 tablet_bootstrap.cc:654] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:26.759266 12874 tablet_bootstrap.cc:654] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:26.759792 12873 log.cc:824] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:26.759919 12874 log.cc:824] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:26.760617 12875 tablet_bootstrap.cc:492] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653: Bootstrap starting.
I20260926 12:33:26.761925 12874 tablet_bootstrap.cc:492] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a: No bootstrap required, opened a new log
I20260926 12:33:26.762058 12873 tablet_bootstrap.cc:492] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527: No bootstrap required, opened a new log
I20260926 12:33:26.762133 12874 ts_tablet_manager.cc:1397] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:33:26.762207 12873 ts_tablet_manager.cc:1397] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:33:26.762619 12875 tablet_bootstrap.cc:654] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:26.763298 12875 log.cc:824] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:26.764870 12875 tablet_bootstrap.cc:492] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653: No bootstrap required, opened a new log
I20260926 12:33:26.765046 12875 ts_tablet_manager.cc:1397] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 12:33:26.766031 12873 raft_consensus.cc:348] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } } peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } }
I20260926 12:33:26.766031 12874 raft_consensus.cc:348] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } } peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } }
I20260926 12:33:26.766314 12873 raft_consensus.cc:374] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:26.766309 12874 raft_consensus.cc:374] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:26.766367 12874 raft_consensus.cc:729] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7caa772a3844ea94caf798f45a9a4a, State: Initialized, Role: FOLLOWER
I20260926 12:33:26.766367 12873 raft_consensus.cc:729] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e485ea0dfa084e81b7c8205293de6527, State: Initialized, Role: FOLLOWER
I20260926 12:33:26.766842 12873 consensus_queue.cc:260] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } } peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } }
I20260926 12:33:26.766842 12874 consensus_queue.cc:260] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } } peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } }
I20260926 12:33:26.767472 12570 heartbeater.cc:503] Master 127.10.98.62:41413 was elected leader, sending a full tablet report...
I20260926 12:33:26.767839 12874 ts_tablet_manager.cc:1428] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:33:26.767840 12873 ts_tablet_manager.cc:1428] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:33:26.768296 12850 heartbeater.cc:503] Master 127.10.98.62:41413 was elected leader, sending a full tablet report...
I20260926 12:33:26.768352 12875 raft_consensus.cc:348] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } } peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } }
I20260926 12:33:26.768610 12875 raft_consensus.cc:374] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:26.768661 12875 raft_consensus.cc:729] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecb0141621ca4a1c9d4f8e7f65f76653, State: Initialized, Role: FOLLOWER
I20260926 12:33:26.769217 12875 consensus_queue.cc:260] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } } peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } }
I20260926 12:33:26.770197 12875 ts_tablet_manager.cc:1428] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:33:26.770517 12710 heartbeater.cc:503] Master 127.10.98.62:41413 was elected leader, sending a full tablet report...
I20260926 12:33:26.823697 12880 raft_consensus.cc:482] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:33:26.823907 12880 raft_consensus.cc:504] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } } peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } }
I20260926 12:33:26.825219 12880 leader_election.cc:288] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecb0141621ca4a1c9d4f8e7f65f76653 (127.10.98.2:42427), e485ea0dfa084e81b7c8205293de6527 (127.10.98.1:41323)
I20260926 12:33:26.829527 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.825825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:47681 (local address 127.10.98.1:41323)
0926 12:33:26.825967 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:26.825972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:26.826037 (+ 65us) server_negotiation.cc:407] Connection header received
0926 12:33:26.826093 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:26.826096 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:26.826145 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:26.826211 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:26.826573 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.827247 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.828438 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.828938 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.829026 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.829082 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:26.829334 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 12:33:26.829392 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 12:33:26.829550 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.825526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:41323 (local address 127.10.98.3:47681)
0926 12:33:26.825654 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:26.825669 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:26.825798 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:26.826251 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:26.826254 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:26.826273 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:26.826459 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.826467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.827347 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.827351 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.828314 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.828323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.829089 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.829092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.829220 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.829246 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 12:33:26.829323 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 12:33:26.829398 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":71}
I20260926 12:33:26.829892 12868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.825618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:50581 (local address 127.10.98.2:42427)
0926 12:33:26.825757 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:26.825762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:26.826073 (+ 311us) server_negotiation.cc:407] Connection header received
0926 12:33:26.826178 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:26.826181 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:26.826233 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:26.826319 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:26.827039 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.827690 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.828787 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.829353 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.829453 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.829508 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:26.829710 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 12:33:26.829783 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:33:26.830214 12882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.825526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:42427 (local address 127.10.98.3:50581)
0926 12:33:26.825946 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:26.825974 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:26.826085 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:26.826337 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:26.826339 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:26.826353 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:26.826913 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.826923 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.827797 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.827800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.828678 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.828688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.829455 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.829458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.829566 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.829590 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:33:26.830024 (+ 434us) client_negotiation.cc:240] Negotiation successful
0926 12:33:26.830097 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":78,"thread_start_us":124,"threads_started":1}
I20260926 12:33:26.830332 12508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f90aa888a6494ea718cc52ed399502" candidate_uuid: "ab7caa772a3844ea94caf798f45a9a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e485ea0dfa084e81b7c8205293de6527" is_pre_election: true
I20260926 12:33:26.830667 12508 raft_consensus.cc:2454] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7caa772a3844ea94caf798f45a9a4a in term 0.
I20260926 12:33:26.830760 12664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f90aa888a6494ea718cc52ed399502" candidate_uuid: "ab7caa772a3844ea94caf798f45a9a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" is_pre_election: true
I20260926 12:33:26.831058 12664 raft_consensus.cc:2454] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7caa772a3844ea94caf798f45a9a4a in term 0.
I20260926 12:33:26.831156 12736 leader_election.cc:302] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab7caa772a3844ea94caf798f45a9a4a, e485ea0dfa084e81b7c8205293de6527; no voters:
I20260926 12:33:26.831430 12880 raft_consensus.cc:2781] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:33:26.831522 12880 raft_consensus.cc:482] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:33:26.831573 12880 raft_consensus.cc:3037] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:26.832551 12880 raft_consensus.cc:504] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } } peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } }
I20260926 12:33:26.832859 12880 leader_election.cc:288] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a [CANDIDATE]: Term 1 election: Requested vote from peers ecb0141621ca4a1c9d4f8e7f65f76653 (127.10.98.2:42427), e485ea0dfa084e81b7c8205293de6527 (127.10.98.1:41323)
I20260926 12:33:26.833231 12664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f90aa888a6494ea718cc52ed399502" candidate_uuid: "ab7caa772a3844ea94caf798f45a9a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653"
I20260926 12:33:26.833361 12664 raft_consensus.cc:3037] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:26.834465 12664 raft_consensus.cc:2454] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7caa772a3844ea94caf798f45a9a4a in term 1.
I20260926 12:33:26.834722 12508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8f90aa888a6494ea718cc52ed399502" candidate_uuid: "ab7caa772a3844ea94caf798f45a9a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e485ea0dfa084e81b7c8205293de6527"
I20260926 12:33:26.834772 12735 leader_election.cc:302] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab7caa772a3844ea94caf798f45a9a4a, ecb0141621ca4a1c9d4f8e7f65f76653; no voters:
I20260926 12:33:26.834838 12508 raft_consensus.cc:3037] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:26.834924 12880 raft_consensus.cc:2781] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:26.835119 12880 raft_consensus.cc:686] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a [term 1 LEADER]: Becoming Leader. State: Replica: ab7caa772a3844ea94caf798f45a9a4a, State: Running, Role: LEADER
I20260926 12:33:26.835461 12880 consensus_queue.cc:237] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } } peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } }
I20260926 12:33:26.836099 12508 raft_consensus.cc:2454] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7caa772a3844ea94caf798f45a9a4a in term 1.
I20260926 12:33:26.838824 12365 catalog_manager.cc:5495] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a reported cstate change: term changed from 0 to 1, leader changed from <none> to ab7caa772a3844ea94caf798f45a9a4a (127.10.98.3). New cstate: current_term: 1 leader_uuid: "ab7caa772a3844ea94caf798f45a9a4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } health_report { overall_health: UNKNOWN } } }
Sep 26 12:33:26 dist-test-slave-t7tk krb5kdc[12322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426005, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:26.891049 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.882913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45846 (local address 127.10.98.62:41413)
0926 12:33:26.883062 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:26.883066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:26.883234 (+ 168us) server_negotiation.cc:407] Connection header received
0926 12:33:26.883347 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:26.883350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:26.883391 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:26.883457 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:26.884120 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.884644 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.885566 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.885781 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.885866 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.888119 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:26.888122 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:26.888124 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:26.888145 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:26.889391 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:26.889907 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:26.889911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:26.889912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:26.889991 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:26.890222 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:26.890224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:26.890225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:26.890405 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:26.890477 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:26.890804 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 12:33:26.890908 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 12:33:26.891058 12890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:26.882822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:41413 (local address 127.0.0.1:45846)
0926 12:33:26.883150 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:26.883163 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:26.883275 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:26.883485 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:26.883487 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:26.883881 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:26.883978 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.883984 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.884756 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.884758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.885445 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:26.885455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:26.885906 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:26.885914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:26.886001 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:26.886497 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:26.886508 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:26.887993 (+ 1485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:26.889507 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:26.889518 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:26.889525 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:26.889770 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:26.890065 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:26.890067 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:26.890069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:26.890153 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:26.890498 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:26.890503 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:26.890626 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 12:33:26.890720 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 12:33:26.890833 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":99,"threads_started":1}
I20260926 12:33:26.894968 12365 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:45846:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 12:33:26.895716 12365 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:33:26.900396 12504 tablet_service.cc:1467] Processing CreateTablet for tablet cf619d6d5f444c178a7155067699824e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=addef24fd96a46aab8532515bc7de894]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:33:26.900570 12644 tablet_service.cc:1467] Processing CreateTablet for tablet cf619d6d5f444c178a7155067699824e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=addef24fd96a46aab8532515bc7de894]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:33:26.901149 12504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf619d6d5f444c178a7155067699824e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:26.901293 12644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf619d6d5f444c178a7155067699824e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:26.903633 12873 tablet_bootstrap.cc:492] T cf619d6d5f444c178a7155067699824e P e485ea0dfa084e81b7c8205293de6527: Bootstrap starting.
I20260926 12:33:26.904062 12784 tablet_service.cc:1467] Processing CreateTablet for tablet cf619d6d5f444c178a7155067699824e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=addef24fd96a46aab8532515bc7de894]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:33:26.904450 12873 tablet_bootstrap.cc:654] T cf619d6d5f444c178a7155067699824e P e485ea0dfa084e81b7c8205293de6527: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:26.904593 12784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf619d6d5f444c178a7155067699824e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:26.904951 12875 tablet_bootstrap.cc:492] T cf619d6d5f444c178a7155067699824e P ecb0141621ca4a1c9d4f8e7f65f76653: Bootstrap starting.
I20260926 12:33:26.905615 12873 tablet_bootstrap.cc:492] T cf619d6d5f444c178a7155067699824e P e485ea0dfa084e81b7c8205293de6527: No bootstrap required, opened a new log
I20260926 12:33:26.905694 12873 ts_tablet_manager.cc:1397] T cf619d6d5f444c178a7155067699824e P e485ea0dfa084e81b7c8205293de6527: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 12:33:26.905892 12875 tablet_bootstrap.cc:654] T cf619d6d5f444c178a7155067699824e P ecb0141621ca4a1c9d4f8e7f65f76653: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:26.906145 12873 raft_consensus.cc:348] T cf619d6d5f444c178a7155067699824e P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:26.906250 12873 raft_consensus.cc:374] T cf619d6d5f444c178a7155067699824e P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:26.906280 12873 raft_consensus.cc:729] T cf619d6d5f444c178a7155067699824e P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e485ea0dfa084e81b7c8205293de6527, State: Initialized, Role: FOLLOWER
I20260926 12:33:26.906405 12873 consensus_queue.cc:260] T cf619d6d5f444c178a7155067699824e P e485ea0dfa084e81b7c8205293de6527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:26.906659 12873 ts_tablet_manager.cc:1428] T cf619d6d5f444c178a7155067699824e P e485ea0dfa084e81b7c8205293de6527: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 12:33:26.906705 12879 tablet_replica.cc:439] T cf619d6d5f444c178a7155067699824e P e485ea0dfa084e81b7c8205293de6527: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } } }
I20260926 12:33:26.906833 12879 tablet_replica.cc:442] T cf619d6d5f444c178a7155067699824e P e485ea0dfa084e81b7c8205293de6527: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:33:26.907083 12874 tablet_bootstrap.cc:492] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a: Bootstrap starting.
I20260926 12:33:26.907665 12875 tablet_bootstrap.cc:492] T cf619d6d5f444c178a7155067699824e P ecb0141621ca4a1c9d4f8e7f65f76653: No bootstrap required, opened a new log
I20260926 12:33:26.907773 12875 ts_tablet_manager.cc:1397] T cf619d6d5f444c178a7155067699824e P ecb0141621ca4a1c9d4f8e7f65f76653: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:33:26.908037 12874 tablet_bootstrap.cc:654] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:26.908241 12875 raft_consensus.cc:348] T cf619d6d5f444c178a7155067699824e P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:26.908351 12875 raft_consensus.cc:374] T cf619d6d5f444c178a7155067699824e P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:26.908389 12875 raft_consensus.cc:729] T cf619d6d5f444c178a7155067699824e P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecb0141621ca4a1c9d4f8e7f65f76653, State: Initialized, Role: FOLLOWER
I20260926 12:33:26.908525 12875 consensus_queue.cc:260] T cf619d6d5f444c178a7155067699824e P ecb0141621ca4a1c9d4f8e7f65f76653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:26.908793 12875 ts_tablet_manager.cc:1428] T cf619d6d5f444c178a7155067699824e P ecb0141621ca4a1c9d4f8e7f65f76653: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:26.909135 12881 tablet_replica.cc:439] T cf619d6d5f444c178a7155067699824e P ecb0141621ca4a1c9d4f8e7f65f76653: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } } }
I20260926 12:33:26.909265 12881 tablet_replica.cc:442] T cf619d6d5f444c178a7155067699824e P ecb0141621ca4a1c9d4f8e7f65f76653: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:33:26.909785 12874 tablet_bootstrap.cc:492] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a: No bootstrap required, opened a new log
I20260926 12:33:26.909853 12874 ts_tablet_manager.cc:1397] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:33:26.910254 12874 raft_consensus.cc:348] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:26.910341 12874 raft_consensus.cc:374] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:26.910368 12874 raft_consensus.cc:729] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7caa772a3844ea94caf798f45a9a4a, State: Initialized, Role: FOLLOWER
I20260926 12:33:26.910485 12874 consensus_queue.cc:260] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:26.910742 12874 ts_tablet_manager.cc:1428] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:26.910734 12880 tablet_replica.cc:439] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } } }
I20260926 12:33:26.910912 12880 tablet_replica.cc:442] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 12:33:26.935264 12851 tablet.cc:2369] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:33:26.967801 12711 tablet.cc:2369] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:33:27.000422 12571 tablet.cc:2369] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:33:27.073765 12880 raft_consensus.cc:482] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:33:27.073904 12880 raft_consensus.cc:504] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:27.074317 12880 leader_election.cc:288] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e485ea0dfa084e81b7c8205293de6527 (127.10.98.1:41323), ecb0141621ca4a1c9d4f8e7f65f76653 (127.10.98.2:42427)
I20260926 12:33:27.074862 12508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf619d6d5f444c178a7155067699824e" candidate_uuid: "ab7caa772a3844ea94caf798f45a9a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e485ea0dfa084e81b7c8205293de6527" is_pre_election: true
I20260926 12:33:27.074890 12664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf619d6d5f444c178a7155067699824e" candidate_uuid: "ab7caa772a3844ea94caf798f45a9a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" is_pre_election: true
I20260926 12:33:27.075017 12508 raft_consensus.cc:2454] T cf619d6d5f444c178a7155067699824e P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7caa772a3844ea94caf798f45a9a4a in term 0.
I20260926 12:33:27.075070 12664 raft_consensus.cc:2454] T cf619d6d5f444c178a7155067699824e P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7caa772a3844ea94caf798f45a9a4a in term 0.
I20260926 12:33:27.075314 12736 leader_election.cc:302] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab7caa772a3844ea94caf798f45a9a4a, e485ea0dfa084e81b7c8205293de6527; no voters:
I20260926 12:33:27.075502 12880 raft_consensus.cc:2781] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:33:27.075582 12880 raft_consensus.cc:482] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:33:27.075613 12880 raft_consensus.cc:3037] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:27.076551 12880 raft_consensus.cc:504] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:27.076874 12880 leader_election.cc:288] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a [CANDIDATE]: Term 1 election: Requested vote from peers e485ea0dfa084e81b7c8205293de6527 (127.10.98.1:41323), ecb0141621ca4a1c9d4f8e7f65f76653 (127.10.98.2:42427)
I20260926 12:33:27.077262 12508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf619d6d5f444c178a7155067699824e" candidate_uuid: "ab7caa772a3844ea94caf798f45a9a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e485ea0dfa084e81b7c8205293de6527"
I20260926 12:33:27.077267 12664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf619d6d5f444c178a7155067699824e" candidate_uuid: "ab7caa772a3844ea94caf798f45a9a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653"
I20260926 12:33:27.077383 12664 raft_consensus.cc:3037] T cf619d6d5f444c178a7155067699824e P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:27.077379 12508 raft_consensus.cc:3037] T cf619d6d5f444c178a7155067699824e P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:27.078287 12664 raft_consensus.cc:2454] T cf619d6d5f444c178a7155067699824e P ecb0141621ca4a1c9d4f8e7f65f76653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7caa772a3844ea94caf798f45a9a4a in term 1.
I20260926 12:33:27.078395 12508 raft_consensus.cc:2454] T cf619d6d5f444c178a7155067699824e P e485ea0dfa084e81b7c8205293de6527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7caa772a3844ea94caf798f45a9a4a in term 1.
I20260926 12:33:27.078552 12735 leader_election.cc:302] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab7caa772a3844ea94caf798f45a9a4a, ecb0141621ca4a1c9d4f8e7f65f76653; no voters:
I20260926 12:33:27.078722 12880 raft_consensus.cc:2781] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:27.078789 12880 raft_consensus.cc:686] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a [term 1 LEADER]: Becoming Leader. State: Replica: ab7caa772a3844ea94caf798f45a9a4a, State: Running, Role: LEADER
I20260926 12:33:27.078922 12880 consensus_queue.cc:237] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:27.079648 12883 tablet_replica.cc:439] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a: TxnStatusTablet state changed. Reason: New leader ab7caa772a3844ea94caf798f45a9a4a. Latest consensus state: current_term: 1 leader_uuid: "ab7caa772a3844ea94caf798f45a9a4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } } }
I20260926 12:33:27.079736 12883 tablet_replica.cc:442] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:33:27.080008 12895 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 12:33:27.080241 12895 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 12:33:27.080746 12365 catalog_manager.cc:5495] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a reported cstate change: term changed from 0 to 1, leader changed from <none> to ab7caa772a3844ea94caf798f45a9a4a (127.10.98.3). New cstate: current_term: 1 leader_uuid: "ab7caa772a3844ea94caf798f45a9a4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260926 12:33:27.100577 12890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:27.096615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:33653 (local address 127.0.0.1:59858)
0926 12:33:27.096756 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:27.096771 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:27.096870 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:27.097194 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:27.097196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:27.097203 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:33:27.097294 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:27.097300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:27.098297 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:27.098300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:27.098894 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:27.098902 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:27.099431 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:27.099434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:27.099519 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:27.099565 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:33:27.100317 (+ 752us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:33:27.100332 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 12:33:27.100384 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 12:33:27.100455 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 12:33:27.100615 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:27.096736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59858 (local address 127.10.98.3:33653)
0926 12:33:27.096863 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:27.096868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:27.096883 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:33:27.096998 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:27.097002 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:27.097062 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:27.097189 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:27.097395 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:27.098179 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:27.098995 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:27.099330 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:27.099427 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:27.099691 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:27.100225 (+ 534us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:27.100333 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:27.100404 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 12:33:27.100465 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:33:27.106042 12508 raft_consensus.cc:1261] T cf619d6d5f444c178a7155067699824e P e485ea0dfa084e81b7c8205293de6527 [term 1 FOLLOWER]: Refusing update from remote peer ab7caa772a3844ea94caf798f45a9a4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:27.106042 12664 raft_consensus.cc:1261] T cf619d6d5f444c178a7155067699824e P ecb0141621ca4a1c9d4f8e7f65f76653 [term 1 FOLLOWER]: Refusing update from remote peer ab7caa772a3844ea94caf798f45a9a4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:27.106779 12880 consensus_queue.cc:1035] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:27.106951 12883 consensus_queue.cc:1035] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:27.110736 12879 tablet_replica.cc:439] T cf619d6d5f444c178a7155067699824e P e485ea0dfa084e81b7c8205293de6527: TxnStatusTablet state changed. Reason: New leader ab7caa772a3844ea94caf798f45a9a4a. Latest consensus state: current_term: 1 leader_uuid: "ab7caa772a3844ea94caf798f45a9a4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } } }
I20260926 12:33:27.110878 12879 tablet_replica.cc:442] T cf619d6d5f444c178a7155067699824e P e485ea0dfa084e81b7c8205293de6527: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:33:27.111307 12881 tablet_replica.cc:439] T cf619d6d5f444c178a7155067699824e P ecb0141621ca4a1c9d4f8e7f65f76653: TxnStatusTablet state changed. Reason: New leader ab7caa772a3844ea94caf798f45a9a4a. Latest consensus state: current_term: 1 leader_uuid: "ab7caa772a3844ea94caf798f45a9a4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } } }
I20260926 12:33:27.111418 12881 tablet_replica.cc:442] T cf619d6d5f444c178a7155067699824e P ecb0141621ca4a1c9d4f8e7f65f76653: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:33:27.115262 12883 tablet_replica.cc:439] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab7caa772a3844ea94caf798f45a9a4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } } }
I20260926 12:33:27.115306 12881 tablet_replica.cc:439] T cf619d6d5f444c178a7155067699824e P ecb0141621ca4a1c9d4f8e7f65f76653: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab7caa772a3844ea94caf798f45a9a4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } } }
I20260926 12:33:27.115396 12883 tablet_replica.cc:442] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:33:27.115417 12881 tablet_replica.cc:442] T cf619d6d5f444c178a7155067699824e P ecb0141621ca4a1c9d4f8e7f65f76653: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:33:27.115453 12899 mvcc.cc:205] Tried to move back new op lower bound from 7333584925099315200 to 7333584924996714496. Current Snapshot: MvccSnapshot[applied={T|T < 7333584925099315200}]
I20260926 12:33:27.115525 12883 tablet_replica.cc:439] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ab7caa772a3844ea94caf798f45a9a4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } } }
I20260926 12:33:27.115594 12883 tablet_replica.cc:442] T cf619d6d5f444c178a7155067699824e P ab7caa772a3844ea94caf798f45a9a4a: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:33:27.115713 12898 mvcc.cc:205] Tried to move back new op lower bound from 7333584925099315200 to 7333584924996714496. Current Snapshot: MvccSnapshot[applied={T|T < 7333584925099315200}]
I20260926 12:33:27.116338 12879 tablet_replica.cc:439] T cf619d6d5f444c178a7155067699824e P e485ea0dfa084e81b7c8205293de6527: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ab7caa772a3844ea94caf798f45a9a4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } } }
I20260926 12:33:27.116446 12879 tablet_replica.cc:442] T cf619d6d5f444c178a7155067699824e P e485ea0dfa084e81b7c8205293de6527: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:33:27.116818 12900 mvcc.cc:205] Tried to move back new op lower bound from 7333584925099315200 to 7333584924996714496. Current Snapshot: MvccSnapshot[applied={T|T < 7333584925099315200}]
I20260926 12:33:27.125362 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:27.122023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59870 (local address 127.10.98.3:33653)
0926 12:33:27.122138 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:27.122143 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:27.122157 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:33:27.122202 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:27.122205 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:27.122247 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:27.122317 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:27.122508 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:27.123279 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:27.124167 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:27.124502 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:27.124596 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:27.124787 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:27.124867 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:27.124945 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:27.125177 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 12:33:27.125229 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 12:33:27.137562 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:27.133798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:33653 (local address 127.10.98.3:52331)
0926 12:33:27.133956 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:27.133972 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:27.134061 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:27.134363 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:27.134365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:27.134373 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:33:27.134473 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:27.134477 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:27.135397 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:27.135399 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:27.136125 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:27.136134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:27.136688 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:27.136691 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:27.136772 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:27.136827 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:33:27.137244 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:33:27.137253 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 12:33:27.137329 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 12:33:27.137433 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":3}
I20260926 12:33:27.137591 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:27.133816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:52331 (local address 127.10.98.3:33653)
0926 12:33:27.134187 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:27.134192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:27.134201 (+ 9us) server_negotiation.cc:407] Connection header received
0926 12:33:27.134233 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:27.134235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:27.134274 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:27.134336 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:27.134562 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:27.135279 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:27.136234 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:27.136562 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:27.136663 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:27.137039 (+ 376us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:27.137161 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:27.137229 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:27.137338 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 12:33:27.137431 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":313}
I20260926 12:33:27.143803 12664 raft_consensus.cc:1261] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 1 FOLLOWER]: Refusing update from remote peer ab7caa772a3844ea94caf798f45a9a4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:27.143862 12508 raft_consensus.cc:1261] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527 [term 1 FOLLOWER]: Refusing update from remote peer ab7caa772a3844ea94caf798f45a9a4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:27.144209 12903 consensus_queue.cc:1035] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:27.144368 12880 consensus_queue.cc:1035] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:27.186702 12365 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:27.186878 12365 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:27.189332 12365 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c: Sending DeleteTablet for 3 replicas of tablet c8f90aa888a6494ea718cc52ed399502
I20260926 12:33:27.190068 12644 tablet_service.cc:1514] Processing DeleteTablet for tablet c8f90aa888a6494ea718cc52ed399502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:27 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:32826
I20260926 12:33:27.190183 12783 tablet_service.cc:1514] Processing DeleteTablet for tablet c8f90aa888a6494ea718cc52ed399502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:27 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:59848
I20260926 12:33:27.190346 12504 tablet_service.cc:1514] Processing DeleteTablet for tablet c8f90aa888a6494ea718cc52ed399502 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:27 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:58766
I20260926 12:33:27.190779 12916 tablet_replica.cc:330] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653: stopping tablet replica
I20260926 12:33:27.190834 12918 tablet_replica.cc:330] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527: stopping tablet replica
I20260926 12:33:27.191174 12916 raft_consensus.cc:2229] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:27.191176 12918 raft_consensus.cc:2229] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:27.191411 12918 raft_consensus.cc:2258] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:27.191533 12916 raft_consensus.cc:2258] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:27.191578 12365 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45836:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:33:27.192003 12365 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:33:27.192198 12917 tablet_replica.cc:330] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a: stopping tablet replica
I20260926 12:33:27.192480 12917 raft_consensus.cc:2229] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:33:27.192812 12917 raft_consensus.cc:2258] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:27.193432 12916 ts_tablet_manager.cc:1905] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:27.193949 12918 ts_tablet_manager.cc:1905] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:27.194284 12917 ts_tablet_manager.cc:1905] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:27.196532 12916 ts_tablet_manager.cc:1918] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:33:27.196623 12916 log.cc:1196] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/c8f90aa888a6494ea718cc52ed399502
I20260926 12:33:27.196775 12918 ts_tablet_manager.cc:1918] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:33:27.196862 12918 log.cc:1196] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-0/wal/wals/c8f90aa888a6494ea718cc52ed399502
I20260926 12:33:27.196946 12916 ts_tablet_manager.cc:1939] T c8f90aa888a6494ea718cc52ed399502 P ecb0141621ca4a1c9d4f8e7f65f76653: Deleting consensus metadata
I20260926 12:33:27.196986 12917 ts_tablet_manager.cc:1918] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:33:27.197065 12917 log.cc:1196] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/c8f90aa888a6494ea718cc52ed399502
I20260926 12:33:27.197196 12504 tablet_service.cc:1467] Processing CreateTablet for tablet ab6787e772e3453f851505664f2999d2 (DEFAULT_TABLE table=test-table [id=0dfa0546d39e48f7930ab1bf67b62b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:27.197360 12917 ts_tablet_manager.cc:1939] T c8f90aa888a6494ea718cc52ed399502 P ab7caa772a3844ea94caf798f45a9a4a: Deleting consensus metadata
I20260926 12:33:27.197430 12504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab6787e772e3453f851505664f2999d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:27.197682 12644 tablet_service.cc:1467] Processing CreateTablet for tablet ab6787e772e3453f851505664f2999d2 (DEFAULT_TABLE table=test-table [id=0dfa0546d39e48f7930ab1bf67b62b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:27.197903 12644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab6787e772e3453f851505664f2999d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:27.198376 12354 catalog_manager.cc:4841] TS ab7caa772a3844ea94caf798f45a9a4a (127.10.98.3:33653): tablet c8f90aa888a6494ea718cc52ed399502 (table test-table [id=f84bcf98e4964a4c9fb5ffe5b5356d91]) successfully deleted
I20260926 12:33:27.198725 12783 tablet_service.cc:1467] Processing CreateTablet for tablet ab6787e772e3453f851505664f2999d2 (DEFAULT_TABLE table=test-table [id=0dfa0546d39e48f7930ab1bf67b62b2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:27.198962 12783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ab6787e772e3453f851505664f2999d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:27.199556 12873 tablet_bootstrap.cc:492] T ab6787e772e3453f851505664f2999d2 P e485ea0dfa084e81b7c8205293de6527: Bootstrap starting.
I20260926 12:33:27.200141 12351 catalog_manager.cc:4841] TS ecb0141621ca4a1c9d4f8e7f65f76653 (127.10.98.2:42427): tablet c8f90aa888a6494ea718cc52ed399502 (table test-table [id=f84bcf98e4964a4c9fb5ffe5b5356d91]) successfully deleted
I20260926 12:33:27.200444 12875 tablet_bootstrap.cc:492] T ab6787e772e3453f851505664f2999d2 P ecb0141621ca4a1c9d4f8e7f65f76653: Bootstrap starting.
I20260926 12:33:27.200490 12918 ts_tablet_manager.cc:1939] T c8f90aa888a6494ea718cc52ed399502 P e485ea0dfa084e81b7c8205293de6527: Deleting consensus metadata
I20260926 12:33:27.201184 12874 tablet_bootstrap.cc:492] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a: Bootstrap starting.
I20260926 12:33:27.201349 12875 tablet_bootstrap.cc:654] T ab6787e772e3453f851505664f2999d2 P ecb0141621ca4a1c9d4f8e7f65f76653: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:27.202076 12874 tablet_bootstrap.cc:654] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:27.202099 12873 tablet_bootstrap.cc:654] T ab6787e772e3453f851505664f2999d2 P e485ea0dfa084e81b7c8205293de6527: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:27.202454 12352 catalog_manager.cc:4841] TS e485ea0dfa084e81b7c8205293de6527 (127.10.98.1:41323): tablet c8f90aa888a6494ea718cc52ed399502 (table test-table [id=f84bcf98e4964a4c9fb5ffe5b5356d91]) successfully deleted
I20260926 12:33:27.203190 12874 tablet_bootstrap.cc:492] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a: No bootstrap required, opened a new log
I20260926 12:33:27.203270 12874 ts_tablet_manager.cc:1397] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 12:33:27.203409 12875 tablet_bootstrap.cc:492] T ab6787e772e3453f851505664f2999d2 P ecb0141621ca4a1c9d4f8e7f65f76653: No bootstrap required, opened a new log
I20260926 12:33:27.203512 12875 ts_tablet_manager.cc:1397] T ab6787e772e3453f851505664f2999d2 P ecb0141621ca4a1c9d4f8e7f65f76653: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:33:27.203711 12874 raft_consensus.cc:348] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:27.203817 12874 raft_consensus.cc:374] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:27.203850 12874 raft_consensus.cc:729] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7caa772a3844ea94caf798f45a9a4a, State: Initialized, Role: FOLLOWER
I20260926 12:33:27.203907 12875 raft_consensus.cc:348] T ab6787e772e3453f851505664f2999d2 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:27.204005 12875 raft_consensus.cc:374] T ab6787e772e3453f851505664f2999d2 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:27.203962 12874 consensus_queue.cc:260] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:27.204033 12875 raft_consensus.cc:729] T ab6787e772e3453f851505664f2999d2 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecb0141621ca4a1c9d4f8e7f65f76653, State: Initialized, Role: FOLLOWER
I20260926 12:33:27.204142 12875 consensus_queue.cc:260] T ab6787e772e3453f851505664f2999d2 P ecb0141621ca4a1c9d4f8e7f65f76653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:27.204205 12874 ts_tablet_manager.cc:1428] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 12:33:27.204368 12875 ts_tablet_manager.cc:1428] T ab6787e772e3453f851505664f2999d2 P ecb0141621ca4a1c9d4f8e7f65f76653: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:27.207864 12873 tablet_bootstrap.cc:492] T ab6787e772e3453f851505664f2999d2 P e485ea0dfa084e81b7c8205293de6527: No bootstrap required, opened a new log
I20260926 12:33:27.207954 12873 ts_tablet_manager.cc:1397] T ab6787e772e3453f851505664f2999d2 P e485ea0dfa084e81b7c8205293de6527: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260926 12:33:27.208377 12873 raft_consensus.cc:348] T ab6787e772e3453f851505664f2999d2 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:27.208520 12873 raft_consensus.cc:374] T ab6787e772e3453f851505664f2999d2 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:27.208557 12873 raft_consensus.cc:729] T ab6787e772e3453f851505664f2999d2 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e485ea0dfa084e81b7c8205293de6527, State: Initialized, Role: FOLLOWER
I20260926 12:33:27.208688 12873 consensus_queue.cc:260] T ab6787e772e3453f851505664f2999d2 P e485ea0dfa084e81b7c8205293de6527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:27.208966 12873 ts_tablet_manager.cc:1428] T ab6787e772e3453f851505664f2999d2 P e485ea0dfa084e81b7c8205293de6527: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:27.238399 12913 raft_consensus.cc:482] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:33:27.238533 12913 raft_consensus.cc:504] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:27.238919 12913 leader_election.cc:288] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e485ea0dfa084e81b7c8205293de6527 (127.10.98.1:41323), ecb0141621ca4a1c9d4f8e7f65f76653 (127.10.98.2:42427)
I20260926 12:33:27.239347 12508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab6787e772e3453f851505664f2999d2" candidate_uuid: "ab7caa772a3844ea94caf798f45a9a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e485ea0dfa084e81b7c8205293de6527" is_pre_election: true
I20260926 12:33:27.239516 12508 raft_consensus.cc:2454] T ab6787e772e3453f851505664f2999d2 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7caa772a3844ea94caf798f45a9a4a in term 0.
I20260926 12:33:27.239537 12664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab6787e772e3453f851505664f2999d2" candidate_uuid: "ab7caa772a3844ea94caf798f45a9a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" is_pre_election: true
I20260926 12:33:27.239696 12664 raft_consensus.cc:2454] T ab6787e772e3453f851505664f2999d2 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7caa772a3844ea94caf798f45a9a4a in term 0.
I20260926 12:33:27.239794 12736 leader_election.cc:302] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab7caa772a3844ea94caf798f45a9a4a, e485ea0dfa084e81b7c8205293de6527; no voters:
I20260926 12:33:27.239965 12913 raft_consensus.cc:2781] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:33:27.240020 12913 raft_consensus.cc:482] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:33:27.240056 12913 raft_consensus.cc:3037] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:27.241006 12913 raft_consensus.cc:504] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:27.241354 12913 leader_election.cc:288] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a [CANDIDATE]: Term 1 election: Requested vote from peers e485ea0dfa084e81b7c8205293de6527 (127.10.98.1:41323), ecb0141621ca4a1c9d4f8e7f65f76653 (127.10.98.2:42427)
I20260926 12:33:27.241663 12664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab6787e772e3453f851505664f2999d2" candidate_uuid: "ab7caa772a3844ea94caf798f45a9a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653"
I20260926 12:33:27.241663 12508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ab6787e772e3453f851505664f2999d2" candidate_uuid: "ab7caa772a3844ea94caf798f45a9a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e485ea0dfa084e81b7c8205293de6527"
I20260926 12:33:27.241773 12508 raft_consensus.cc:3037] T ab6787e772e3453f851505664f2999d2 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:27.241772 12664 raft_consensus.cc:3037] T ab6787e772e3453f851505664f2999d2 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:27.242611 12664 raft_consensus.cc:2454] T ab6787e772e3453f851505664f2999d2 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7caa772a3844ea94caf798f45a9a4a in term 1.
I20260926 12:33:27.242878 12735 leader_election.cc:302] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab7caa772a3844ea94caf798f45a9a4a, ecb0141621ca4a1c9d4f8e7f65f76653; no voters:
I20260926 12:33:27.242925 12508 raft_consensus.cc:2454] T ab6787e772e3453f851505664f2999d2 P e485ea0dfa084e81b7c8205293de6527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7caa772a3844ea94caf798f45a9a4a in term 1.
I20260926 12:33:27.243036 12913 raft_consensus.cc:2781] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:27.243119 12913 raft_consensus.cc:686] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a [term 1 LEADER]: Becoming Leader. State: Replica: ab7caa772a3844ea94caf798f45a9a4a, State: Running, Role: LEADER
I20260926 12:33:27.243279 12913 consensus_queue.cc:237] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:27.245051 12367 catalog_manager.cc:5495] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a reported cstate change: term changed from 0 to 1, leader changed from <none> to ab7caa772a3844ea94caf798f45a9a4a (127.10.98.3). New cstate: current_term: 1 leader_uuid: "ab7caa772a3844ea94caf798f45a9a4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260926 12:33:27.272154 12664 raft_consensus.cc:1261] T ab6787e772e3453f851505664f2999d2 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 1 FOLLOWER]: Refusing update from remote peer ab7caa772a3844ea94caf798f45a9a4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:27.272379 12508 raft_consensus.cc:1261] T ab6787e772e3453f851505664f2999d2 P e485ea0dfa084e81b7c8205293de6527 [term 1 FOLLOWER]: Refusing update from remote peer ab7caa772a3844ea94caf798f45a9a4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:27.272552 12883 consensus_queue.cc:1035] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:27.273015 12883 consensus_queue.cc:1035] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 12:33:27.287351 12898 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 12:33:28.836724 12365 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:28.836993 12365 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45836:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:28.839468 12365 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c: Sending DeleteTablet for 3 replicas of tablet ab6787e772e3453f851505664f2999d2
I20260926 12:33:28.840041 12504 tablet_service.cc:1514] Processing DeleteTablet for tablet ab6787e772e3453f851505664f2999d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:28 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:58766
I20260926 12:33:28.840052 12784 tablet_service.cc:1514] Processing DeleteTablet for tablet ab6787e772e3453f851505664f2999d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:28 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:59848
I20260926 12:33:28.840076 12644 tablet_service.cc:1514] Processing DeleteTablet for tablet ab6787e772e3453f851505664f2999d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:28 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:32826
I20260926 12:33:28.840373 12946 tablet_replica.cc:330] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a: stopping tablet replica
I20260926 12:33:28.840372 12945 tablet_replica.cc:330] T ab6787e772e3453f851505664f2999d2 P e485ea0dfa084e81b7c8205293de6527: stopping tablet replica
I20260926 12:33:28.840502 12946 raft_consensus.cc:2229] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:33:28.840502 12945 raft_consensus.cc:2229] T ab6787e772e3453f851505664f2999d2 P e485ea0dfa084e81b7c8205293de6527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:28.840627 12945 raft_consensus.cc:2258] T ab6787e772e3453f851505664f2999d2 P e485ea0dfa084e81b7c8205293de6527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:28.840852 12944 tablet_replica.cc:330] T ab6787e772e3453f851505664f2999d2 P ecb0141621ca4a1c9d4f8e7f65f76653: stopping tablet replica
I20260926 12:33:28.840700 12946 raft_consensus.cc:2258] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:28.841076 12944 raft_consensus.cc:2229] T ab6787e772e3453f851505664f2999d2 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:28.841245 12944 raft_consensus.cc:2258] T ab6787e772e3453f851505664f2999d2 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:28.841298 12945 ts_tablet_manager.cc:1905] T ab6787e772e3453f851505664f2999d2 P e485ea0dfa084e81b7c8205293de6527: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:28.841637 12946 ts_tablet_manager.cc:1905] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:28.841794 12944 ts_tablet_manager.cc:1905] T ab6787e772e3453f851505664f2999d2 P ecb0141621ca4a1c9d4f8e7f65f76653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:28.844241 12944 ts_tablet_manager.cc:1918] T ab6787e772e3453f851505664f2999d2 P ecb0141621ca4a1c9d4f8e7f65f76653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 12:33:28.844241 12946 ts_tablet_manager.cc:1918] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 12:33:28.844300 12945 ts_tablet_manager.cc:1918] T ab6787e772e3453f851505664f2999d2 P e485ea0dfa084e81b7c8205293de6527: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 12:33:28.844354 12944 log.cc:1196] T ab6787e772e3453f851505664f2999d2 P ecb0141621ca4a1c9d4f8e7f65f76653: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/ab6787e772e3453f851505664f2999d2
I20260926 12:33:28.844362 12946 log.cc:1196] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/ab6787e772e3453f851505664f2999d2
I20260926 12:33:28.844432 12945 log.cc:1196] T ab6787e772e3453f851505664f2999d2 P e485ea0dfa084e81b7c8205293de6527: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-0/wal/wals/ab6787e772e3453f851505664f2999d2
I20260926 12:33:28.844717 12944 ts_tablet_manager.cc:1939] T ab6787e772e3453f851505664f2999d2 P ecb0141621ca4a1c9d4f8e7f65f76653: Deleting consensus metadata
I20260926 12:33:28.844720 12945 ts_tablet_manager.cc:1939] T ab6787e772e3453f851505664f2999d2 P e485ea0dfa084e81b7c8205293de6527: Deleting consensus metadata
I20260926 12:33:28.844712 12946 ts_tablet_manager.cc:1939] T ab6787e772e3453f851505664f2999d2 P ab7caa772a3844ea94caf798f45a9a4a: Deleting consensus metadata
I20260926 12:33:28.845448 12352 catalog_manager.cc:4841] TS e485ea0dfa084e81b7c8205293de6527 (127.10.98.1:41323): tablet ab6787e772e3453f851505664f2999d2 (table test-table [id=0dfa0546d39e48f7930ab1bf67b62b2a]) successfully deleted
I20260926 12:33:28.845535 12351 catalog_manager.cc:4841] TS ecb0141621ca4a1c9d4f8e7f65f76653 (127.10.98.2:42427): tablet ab6787e772e3453f851505664f2999d2 (table test-table [id=0dfa0546d39e48f7930ab1bf67b62b2a]) successfully deleted
I20260926 12:33:28.845714 12354 catalog_manager.cc:4841] TS ab7caa772a3844ea94caf798f45a9a4a (127.10.98.3:33653): tablet ab6787e772e3453f851505664f2999d2 (table test-table [id=0dfa0546d39e48f7930ab1bf67b62b2a]) successfully deleted
Sep 26 12:33:28 dist-test-slave-t7tk krb5kdc[12322](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426008, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 12:33:28 dist-test-slave-t7tk krb5kdc[12322](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426008, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:28.868505 12954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:28.859585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46226 (local address 127.10.98.62:41413)
0926 12:33:28.859839 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:28.859846 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:28.859860 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:33:28.859904 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:28.859906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:28.859949 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:28.860085 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:28.860883 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:28.861542 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:28.862411 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:28.862671 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:28.862788 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:28.865380 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:28.865383 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:28.865385 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:28.865414 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:28.866866 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:28.867282 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:28.867286 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:28.867288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:28.867370 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:28.867629 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:28.867633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:28.867635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:28.867827 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:28.867949 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:28.868197 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 12:33:28.868325 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":89,"threads_started":1}
I20260926 12:33:28.871090 12365 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:33:28.871556 12365 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:33:28.876520 12504 tablet_service.cc:1467] Processing CreateTablet for tablet ae406a79b17d430c8c2c13643a06f831 (DEFAULT_TABLE table=test-table [id=ca35b85d5e5f4fa191ed48ff44c70e2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:28.876520 12644 tablet_service.cc:1467] Processing CreateTablet for tablet ae406a79b17d430c8c2c13643a06f831 (DEFAULT_TABLE table=test-table [id=ca35b85d5e5f4fa191ed48ff44c70e2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:28.876587 12784 tablet_service.cc:1467] Processing CreateTablet for tablet ae406a79b17d430c8c2c13643a06f831 (DEFAULT_TABLE table=test-table [id=ca35b85d5e5f4fa191ed48ff44c70e2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:28.876829 12504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae406a79b17d430c8c2c13643a06f831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:28.876839 12784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae406a79b17d430c8c2c13643a06f831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:28.876834 12644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae406a79b17d430c8c2c13643a06f831. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:28.879081 12955 tablet_bootstrap.cc:492] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653: Bootstrap starting.
I20260926 12:33:28.879761 12957 tablet_bootstrap.cc:492] T ae406a79b17d430c8c2c13643a06f831 P ab7caa772a3844ea94caf798f45a9a4a: Bootstrap starting.
I20260926 12:33:28.879997 12955 tablet_bootstrap.cc:654] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:28.880460 12957 tablet_bootstrap.cc:654] T ae406a79b17d430c8c2c13643a06f831 P ab7caa772a3844ea94caf798f45a9a4a: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:28.881078 12956 tablet_bootstrap.cc:492] T ae406a79b17d430c8c2c13643a06f831 P e485ea0dfa084e81b7c8205293de6527: Bootstrap starting.
I20260926 12:33:28.881513 12955 tablet_bootstrap.cc:492] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653: No bootstrap required, opened a new log
I20260926 12:33:28.881659 12955 ts_tablet_manager.cc:1397] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:33:28.881675 12957 tablet_bootstrap.cc:492] T ae406a79b17d430c8c2c13643a06f831 P ab7caa772a3844ea94caf798f45a9a4a: No bootstrap required, opened a new log
I20260926 12:33:28.881789 12957 ts_tablet_manager.cc:1397] T ae406a79b17d430c8c2c13643a06f831 P ab7caa772a3844ea94caf798f45a9a4a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 12:33:28.882205 12955 raft_consensus.cc:348] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:28.882296 12956 tablet_bootstrap.cc:654] T ae406a79b17d430c8c2c13643a06f831 P e485ea0dfa084e81b7c8205293de6527: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:28.882370 12955 raft_consensus.cc:374] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:28.882257 12957 raft_consensus.cc:348] T ae406a79b17d430c8c2c13643a06f831 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:28.882422 12955 raft_consensus.cc:729] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecb0141621ca4a1c9d4f8e7f65f76653, State: Initialized, Role: FOLLOWER
I20260926 12:33:28.882431 12957 raft_consensus.cc:374] T ae406a79b17d430c8c2c13643a06f831 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:28.882467 12957 raft_consensus.cc:729] T ae406a79b17d430c8c2c13643a06f831 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7caa772a3844ea94caf798f45a9a4a, State: Initialized, Role: FOLLOWER
I20260926 12:33:28.882575 12955 consensus_queue.cc:260] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:28.882640 12957 consensus_queue.cc:260] T ae406a79b17d430c8c2c13643a06f831 P ab7caa772a3844ea94caf798f45a9a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:28.882956 12957 ts_tablet_manager.cc:1428] T ae406a79b17d430c8c2c13643a06f831 P ab7caa772a3844ea94caf798f45a9a4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:28.882958 12955 ts_tablet_manager.cc:1428] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:28.883476 12956 tablet_bootstrap.cc:492] T ae406a79b17d430c8c2c13643a06f831 P e485ea0dfa084e81b7c8205293de6527: No bootstrap required, opened a new log
I20260926 12:33:28.883577 12956 ts_tablet_manager.cc:1397] T ae406a79b17d430c8c2c13643a06f831 P e485ea0dfa084e81b7c8205293de6527: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 12:33:28.884034 12956 raft_consensus.cc:348] T ae406a79b17d430c8c2c13643a06f831 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:28.884158 12956 raft_consensus.cc:374] T ae406a79b17d430c8c2c13643a06f831 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:28.884188 12956 raft_consensus.cc:729] T ae406a79b17d430c8c2c13643a06f831 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e485ea0dfa084e81b7c8205293de6527, State: Initialized, Role: FOLLOWER
I20260926 12:33:28.884305 12956 consensus_queue.cc:260] T ae406a79b17d430c8c2c13643a06f831 P e485ea0dfa084e81b7c8205293de6527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:28.884684 12956 ts_tablet_manager.cc:1428] T ae406a79b17d430c8c2c13643a06f831 P e485ea0dfa084e81b7c8205293de6527: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 12:33:29.022725 12960 raft_consensus.cc:482] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:33:29.022897 12960 raft_consensus.cc:504] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:29.024040 12960 leader_election.cc:288] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e485ea0dfa084e81b7c8205293de6527 (127.10.98.1:41323), ab7caa772a3844ea94caf798f45a9a4a (127.10.98.3:33653)
I20260926 12:33:29.029603 12966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:29.024493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:42787 (local address 127.10.98.1:41323)
0926 12:33:29.024785 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:29.024794 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:29.024910 (+ 116us) server_negotiation.cc:407] Connection header received
0926 12:33:29.025124 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:29.025127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:29.025173 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:29.025252 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:29.025677 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.026510 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.028316 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.028919 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.029037 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:29.029130 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:29.029351 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 12:33:29.029432 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":102,"threads_started":1}
I20260926 12:33:29.029626 12964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:29.024345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:41323 (local address 127.10.98.2:42787)
0926 12:33:29.024807 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:29.024825 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:29.024962 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:29.025288 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:29.025292 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:29.025365 (+ 73us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:29.025544 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:29.025553 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.026639 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.026643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:29.028149 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:29.028159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.029060 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.029064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:29.029202 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:29.029231 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 12:33:29.029316 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 12:33:29.029412 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":106,"spinlock_wait_cycles":10624,"thread_start_us":159,"threads_started":1}
I20260926 12:33:29.029639 12963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:29.024368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:33653 (local address 127.10.98.2:46949)
0926 12:33:29.024859 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:29.024871 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:29.024963 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:29.025288 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:29.025291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:29.025363 (+ 72us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:29.025560 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:29.025566 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.026639 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.026643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:29.028147 (+ 1504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:29.028159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.029234 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.029238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:29.029334 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:29.029356 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:33:29.029420 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 12:33:29.029490 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":440,"mutex_wait_us":95,"spinlock_wait_cycles":106368,"thread_start_us":158,"threads_started":1}
I20260926 12:33:29.029906 12965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:29.024492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:46949 (local address 127.10.98.3:33653)
0926 12:33:29.024785 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:29.024794 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:29.024939 (+ 145us) server_negotiation.cc:407] Connection header received
0926 12:33:29.025070 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:29.025073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:29.025154 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:29.025249 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:29.025677 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.026488 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.028476 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.029123 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.029218 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:29.029278 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:29.029690 (+ 412us) server_negotiation.cc:299] Negotiation successful
0926 12:33:29.029780 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":93,"threads_started":1}
I20260926 12:33:29.030176 12524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae406a79b17d430c8c2c13643a06f831" candidate_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e485ea0dfa084e81b7c8205293de6527" is_pre_election: true
I20260926 12:33:29.030402 12524 raft_consensus.cc:2454] T ae406a79b17d430c8c2c13643a06f831 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecb0141621ca4a1c9d4f8e7f65f76653 in term 0.
I20260926 12:33:29.030460 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae406a79b17d430c8c2c13643a06f831" candidate_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7caa772a3844ea94caf798f45a9a4a" is_pre_election: true
I20260926 12:33:29.030699 12804 raft_consensus.cc:2454] T ae406a79b17d430c8c2c13643a06f831 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecb0141621ca4a1c9d4f8e7f65f76653 in term 0.
I20260926 12:33:29.030778 12596 leader_election.cc:302] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e485ea0dfa084e81b7c8205293de6527, ecb0141621ca4a1c9d4f8e7f65f76653; no voters:
I20260926 12:33:29.031054 12960 raft_consensus.cc:2781] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:33:29.031111 12960 raft_consensus.cc:482] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:33:29.031152 12960 raft_consensus.cc:3037] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:29.032171 12960 raft_consensus.cc:504] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:29.032529 12960 leader_election.cc:288] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 [CANDIDATE]: Term 1 election: Requested vote from peers e485ea0dfa084e81b7c8205293de6527 (127.10.98.1:41323), ab7caa772a3844ea94caf798f45a9a4a (127.10.98.3:33653)
I20260926 12:33:29.032900 12524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae406a79b17d430c8c2c13643a06f831" candidate_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e485ea0dfa084e81b7c8205293de6527"
I20260926 12:33:29.032912 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae406a79b17d430c8c2c13643a06f831" candidate_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7caa772a3844ea94caf798f45a9a4a"
I20260926 12:33:29.033013 12804 raft_consensus.cc:3037] T ae406a79b17d430c8c2c13643a06f831 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:29.033026 12524 raft_consensus.cc:3037] T ae406a79b17d430c8c2c13643a06f831 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:29.033967 12804 raft_consensus.cc:2454] T ae406a79b17d430c8c2c13643a06f831 P ab7caa772a3844ea94caf798f45a9a4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecb0141621ca4a1c9d4f8e7f65f76653 in term 1.
I20260926 12:33:29.033989 12524 raft_consensus.cc:2454] T ae406a79b17d430c8c2c13643a06f831 P e485ea0dfa084e81b7c8205293de6527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecb0141621ca4a1c9d4f8e7f65f76653 in term 1.
I20260926 12:33:29.034255 12598 leader_election.cc:302] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab7caa772a3844ea94caf798f45a9a4a, ecb0141621ca4a1c9d4f8e7f65f76653; no voters:
I20260926 12:33:29.034421 12960 raft_consensus.cc:2781] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:29.034636 12960 raft_consensus.cc:686] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 1 LEADER]: Becoming Leader. State: Replica: ecb0141621ca4a1c9d4f8e7f65f76653, State: Running, Role: LEADER
I20260926 12:33:29.034919 12960 consensus_queue.cc:237] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:29.037465 12365 catalog_manager.cc:5495] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 reported cstate change: term changed from 0 to 1, leader changed from <none> to ecb0141621ca4a1c9d4f8e7f65f76653 (127.10.98.2). New cstate: current_term: 1 leader_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } health_report { overall_health: UNKNOWN } } }
I20260926 12:33:29.079594 12970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:29.075847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42834 (local address 127.10.98.2:42427)
0926 12:33:29.076150 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:29.076158 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:29.076170 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:33:29.076238 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:29.076240 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:29.076290 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:29.076414 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:29.076588 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.077275 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.078120 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.078366 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.078463 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:29.078678 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:29.079223 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:29.079331 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:29.079396 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 12:33:29.079453 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":165,"threads_started":1}
I20260926 12:33:29.090926 12973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:29.087211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:33653 (local address 127.10.98.2:42541)
0926 12:33:29.087602 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:29.087615 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:29.087723 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:29.088170 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:29.088173 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:29.088182 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:33:29.088298 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:29.088303 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.089043 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.089047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:29.089763 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:29.089773 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.090194 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.090197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:29.090290 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:29.090339 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:33:29.090619 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:33:29.090625 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 12:33:29.090684 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 12:33:29.090767 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":163,"threads_started":1}
I20260926 12:33:29.090914 12965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:29.087390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:42541 (local address 127.10.98.3:33653)
0926 12:33:29.087566 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:29.087571 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:29.087678 (+ 107us) server_negotiation.cc:407] Connection header received
0926 12:33:29.087819 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:29.087822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:29.087992 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:29.088150 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:29.088398 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.088927 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.089871 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.090092 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.090184 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:29.090462 (+ 278us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:29.090542 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:29.090599 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:29.090693 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 12:33:29.090767 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 12:33:29.097163 12524 raft_consensus.cc:1261] T ae406a79b17d430c8c2c13643a06f831 P e485ea0dfa084e81b7c8205293de6527 [term 1 FOLLOWER]: Refusing update from remote peer ecb0141621ca4a1c9d4f8e7f65f76653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:29.097307 12804 raft_consensus.cc:1261] T ae406a79b17d430c8c2c13643a06f831 P ab7caa772a3844ea94caf798f45a9a4a [term 1 FOLLOWER]: Refusing update from remote peer ecb0141621ca4a1c9d4f8e7f65f76653: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:29.097687 12960 consensus_queue.cc:1035] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:29.097872 12967 consensus_queue.cc:1035] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:29.114099 12970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:29.110830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:46939 (local address 127.10.98.2:42427)
0926 12:33:29.110934 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:29.110943 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:29.111118 (+ 175us) server_negotiation.cc:407] Connection header received
0926 12:33:29.111223 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:29.111226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:29.111271 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:29.111336 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:29.111564 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.112064 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.112976 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.113296 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.113377 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:29.113649 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:29.113716 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:29.113767 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:29.113889 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 12:33:29.113955 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 12:33:29.114151 12978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:29.110773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:42427 (local address 127.10.98.3:46939)
0926 12:33:29.111040 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:29.111051 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:29.111145 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:29.111354 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:29.111357 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:29.111367 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:33:29.111491 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:29.111496 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.112168 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.112171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:29.112806 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:29.112816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.113402 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.113406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:29.113509 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:29.113544 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:33:29.113787 (+ 243us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:33:29.113793 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 12:33:29.113860 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 12:33:29.114005 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":118,"threads_started":1}
I20260926 12:33:29.138438 12365 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:29.138599 12365 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:29.140836 12365 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c: Sending DeleteTablet for 3 replicas of tablet ae406a79b17d430c8c2c13643a06f831
I20260926 12:33:29.141485 12504 tablet_service.cc:1514] Processing DeleteTablet for tablet ae406a79b17d430c8c2c13643a06f831 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:29 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:58766
I20260926 12:33:29.141680 12945 tablet_replica.cc:330] T ae406a79b17d430c8c2c13643a06f831 P e485ea0dfa084e81b7c8205293de6527: stopping tablet replica
I20260926 12:33:29.141654 12784 tablet_service.cc:1514] Processing DeleteTablet for tablet ae406a79b17d430c8c2c13643a06f831 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:29 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:59848
I20260926 12:33:29.141796 12945 raft_consensus.cc:2229] T ae406a79b17d430c8c2c13643a06f831 P e485ea0dfa084e81b7c8205293de6527 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:29.141788 12644 tablet_service.cc:1514] Processing DeleteTablet for tablet ae406a79b17d430c8c2c13643a06f831 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:29 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:32826
I20260926 12:33:29.141913 12945 raft_consensus.cc:2258] T ae406a79b17d430c8c2c13643a06f831 P e485ea0dfa084e81b7c8205293de6527 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:29.141985 12944 tablet_replica.cc:330] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653: stopping tablet replica
I20260926 12:33:29.142014 12946 tablet_replica.cc:330] T ae406a79b17d430c8c2c13643a06f831 P ab7caa772a3844ea94caf798f45a9a4a: stopping tablet replica
I20260926 12:33:29.142086 12944 raft_consensus.cc:2229] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:33:29.142143 12946 raft_consensus.cc:2229] T ae406a79b17d430c8c2c13643a06f831 P ab7caa772a3844ea94caf798f45a9a4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:29.142254 12946 raft_consensus.cc:2258] T ae406a79b17d430c8c2c13643a06f831 P ab7caa772a3844ea94caf798f45a9a4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:29.142400 12944 raft_consensus.cc:2258] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:29.142640 12945 ts_tablet_manager.cc:1905] T ae406a79b17d430c8c2c13643a06f831 P e485ea0dfa084e81b7c8205293de6527: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:29.143340 12946 ts_tablet_manager.cc:1905] T ae406a79b17d430c8c2c13643a06f831 P ab7caa772a3844ea94caf798f45a9a4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:29.144156 12365 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46226:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:33:29.144618 12365 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:33:29.145663 12944 ts_tablet_manager.cc:1905] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:29.145887 12946 ts_tablet_manager.cc:1918] T ae406a79b17d430c8c2c13643a06f831 P ab7caa772a3844ea94caf798f45a9a4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:33:29.145951 12946 log.cc:1196] T ae406a79b17d430c8c2c13643a06f831 P ab7caa772a3844ea94caf798f45a9a4a: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/ae406a79b17d430c8c2c13643a06f831
I20260926 12:33:29.146159 12946 ts_tablet_manager.cc:1939] T ae406a79b17d430c8c2c13643a06f831 P ab7caa772a3844ea94caf798f45a9a4a: Deleting consensus metadata
I20260926 12:33:29.146849 12945 ts_tablet_manager.cc:1918] T ae406a79b17d430c8c2c13643a06f831 P e485ea0dfa084e81b7c8205293de6527: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:33:29.146936 12945 log.cc:1196] T ae406a79b17d430c8c2c13643a06f831 P e485ea0dfa084e81b7c8205293de6527: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-0/wal/wals/ae406a79b17d430c8c2c13643a06f831
I20260926 12:33:29.147171 12945 ts_tablet_manager.cc:1939] T ae406a79b17d430c8c2c13643a06f831 P e485ea0dfa084e81b7c8205293de6527: Deleting consensus metadata
I20260926 12:33:29.148386 12352 catalog_manager.cc:4841] TS e485ea0dfa084e81b7c8205293de6527 (127.10.98.1:41323): tablet ae406a79b17d430c8c2c13643a06f831 (table test-table [id=ca35b85d5e5f4fa191ed48ff44c70e2f]) successfully deleted
I20260926 12:33:29.148854 12354 catalog_manager.cc:4841] TS ab7caa772a3844ea94caf798f45a9a4a (127.10.98.3:33653): tablet ae406a79b17d430c8c2c13643a06f831 (table test-table [id=ca35b85d5e5f4fa191ed48ff44c70e2f]) successfully deleted
I20260926 12:33:29.149591 12944 ts_tablet_manager.cc:1918] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:33:29.149765 12944 log.cc:1196] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/ae406a79b17d430c8c2c13643a06f831
I20260926 12:33:29.150029 12944 ts_tablet_manager.cc:1939] T ae406a79b17d430c8c2c13643a06f831 P ecb0141621ca4a1c9d4f8e7f65f76653: Deleting consensus metadata
I20260926 12:33:29.151124 12504 tablet_service.cc:1467] Processing CreateTablet for tablet 5bbcf21d051041c8a0691db6c9723ae2 (DEFAULT_TABLE table=test-table [id=2034f845f46b48fc95d5e1a3d02b821b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:29.151381 12504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bbcf21d051041c8a0691db6c9723ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:29.151374 12784 tablet_service.cc:1467] Processing CreateTablet for tablet 5bbcf21d051041c8a0691db6c9723ae2 (DEFAULT_TABLE table=test-table [id=2034f845f46b48fc95d5e1a3d02b821b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:29.151376 12644 tablet_service.cc:1467] Processing CreateTablet for tablet 5bbcf21d051041c8a0691db6c9723ae2 (DEFAULT_TABLE table=test-table [id=2034f845f46b48fc95d5e1a3d02b821b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:29.151629 12644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bbcf21d051041c8a0691db6c9723ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:29.151629 12784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bbcf21d051041c8a0691db6c9723ae2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:29.153826 12956 tablet_bootstrap.cc:492] T 5bbcf21d051041c8a0691db6c9723ae2 P e485ea0dfa084e81b7c8205293de6527: Bootstrap starting.
I20260926 12:33:29.154350 12957 tablet_bootstrap.cc:492] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a: Bootstrap starting.
I20260926 12:33:29.154755 12956 tablet_bootstrap.cc:654] T 5bbcf21d051041c8a0691db6c9723ae2 P e485ea0dfa084e81b7c8205293de6527: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:29.155030 12957 tablet_bootstrap.cc:654] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:29.155758 12955 tablet_bootstrap.cc:492] T 5bbcf21d051041c8a0691db6c9723ae2 P ecb0141621ca4a1c9d4f8e7f65f76653: Bootstrap starting.
I20260926 12:33:29.156122 12956 tablet_bootstrap.cc:492] T 5bbcf21d051041c8a0691db6c9723ae2 P e485ea0dfa084e81b7c8205293de6527: No bootstrap required, opened a new log
I20260926 12:33:29.156204 12956 ts_tablet_manager.cc:1397] T 5bbcf21d051041c8a0691db6c9723ae2 P e485ea0dfa084e81b7c8205293de6527: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 12:33:29.156590 12351 catalog_manager.cc:4841] TS ecb0141621ca4a1c9d4f8e7f65f76653 (127.10.98.2:42427): tablet ae406a79b17d430c8c2c13643a06f831 (table test-table [id=ca35b85d5e5f4fa191ed48ff44c70e2f]) successfully deleted
I20260926 12:33:29.156659 12955 tablet_bootstrap.cc:654] T 5bbcf21d051041c8a0691db6c9723ae2 P ecb0141621ca4a1c9d4f8e7f65f76653: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:29.156649 12956 raft_consensus.cc:348] T 5bbcf21d051041c8a0691db6c9723ae2 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:29.156761 12956 raft_consensus.cc:374] T 5bbcf21d051041c8a0691db6c9723ae2 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:29.156795 12956 raft_consensus.cc:729] T 5bbcf21d051041c8a0691db6c9723ae2 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e485ea0dfa084e81b7c8205293de6527, State: Initialized, Role: FOLLOWER
I20260926 12:33:29.156925 12956 consensus_queue.cc:260] T 5bbcf21d051041c8a0691db6c9723ae2 P e485ea0dfa084e81b7c8205293de6527 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:29.157166 12957 tablet_bootstrap.cc:492] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a: No bootstrap required, opened a new log
I20260926 12:33:29.157227 12956 ts_tablet_manager.cc:1428] T 5bbcf21d051041c8a0691db6c9723ae2 P e485ea0dfa084e81b7c8205293de6527: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:29.157256 12957 ts_tablet_manager.cc:1397] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:33:29.157729 12957 raft_consensus.cc:348] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:29.157868 12957 raft_consensus.cc:374] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:29.157914 12957 raft_consensus.cc:729] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7caa772a3844ea94caf798f45a9a4a, State: Initialized, Role: FOLLOWER
I20260926 12:33:29.158051 12957 consensus_queue.cc:260] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:29.158422 12957 ts_tablet_manager.cc:1428] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:29.158458 12955 tablet_bootstrap.cc:492] T 5bbcf21d051041c8a0691db6c9723ae2 P ecb0141621ca4a1c9d4f8e7f65f76653: No bootstrap required, opened a new log
I20260926 12:33:29.158535 12955 ts_tablet_manager.cc:1397] T 5bbcf21d051041c8a0691db6c9723ae2 P ecb0141621ca4a1c9d4f8e7f65f76653: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:33:29.158918 12955 raft_consensus.cc:348] T 5bbcf21d051041c8a0691db6c9723ae2 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:29.159011 12955 raft_consensus.cc:374] T 5bbcf21d051041c8a0691db6c9723ae2 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:29.159039 12955 raft_consensus.cc:729] T 5bbcf21d051041c8a0691db6c9723ae2 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecb0141621ca4a1c9d4f8e7f65f76653, State: Initialized, Role: FOLLOWER
I20260926 12:33:29.159160 12955 consensus_queue.cc:260] T 5bbcf21d051041c8a0691db6c9723ae2 P ecb0141621ca4a1c9d4f8e7f65f76653 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:29.159417 12955 ts_tablet_manager.cc:1428] T 5bbcf21d051041c8a0691db6c9723ae2 P ecb0141621ca4a1c9d4f8e7f65f76653: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:29.174036 12979 raft_consensus.cc:482] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:33:29.174194 12979 raft_consensus.cc:504] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:29.174374 12962 raft_consensus.cc:482] T 5bbcf21d051041c8a0691db6c9723ae2 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:33:29.174500 12962 raft_consensus.cc:504] T 5bbcf21d051041c8a0691db6c9723ae2 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:29.174814 12979 leader_election.cc:288] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e485ea0dfa084e81b7c8205293de6527 (127.10.98.1:41323), ecb0141621ca4a1c9d4f8e7f65f76653 (127.10.98.2:42427)
I20260926 12:33:29.175169 12524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bbcf21d051041c8a0691db6c9723ae2" candidate_uuid: "ab7caa772a3844ea94caf798f45a9a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e485ea0dfa084e81b7c8205293de6527" is_pre_election: true
I20260926 12:33:29.175287 12524 raft_consensus.cc:2454] T 5bbcf21d051041c8a0691db6c9723ae2 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7caa772a3844ea94caf798f45a9a4a in term 0.
I20260926 12:33:29.175789 12736 leader_election.cc:302] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab7caa772a3844ea94caf798f45a9a4a, e485ea0dfa084e81b7c8205293de6527; no voters:
I20260926 12:33:29.175927 12962 leader_election.cc:288] T 5bbcf21d051041c8a0691db6c9723ae2 P e485ea0dfa084e81b7c8205293de6527 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecb0141621ca4a1c9d4f8e7f65f76653 (127.10.98.2:42427), ab7caa772a3844ea94caf798f45a9a4a (127.10.98.3:33653)
I20260926 12:33:29.175971 12979 raft_consensus.cc:2781] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:33:29.176030 12979 raft_consensus.cc:482] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:33:29.176064 12979 raft_consensus.cc:3037] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:29.176662 12664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bbcf21d051041c8a0691db6c9723ae2" candidate_uuid: "ab7caa772a3844ea94caf798f45a9a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" is_pre_election: true
I20260926 12:33:29.176808 12664 raft_consensus.cc:2454] T 5bbcf21d051041c8a0691db6c9723ae2 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab7caa772a3844ea94caf798f45a9a4a in term 0.
I20260926 12:33:29.177295 12979 raft_consensus.cc:504] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:29.177661 12979 leader_election.cc:288] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a [CANDIDATE]: Term 1 election: Requested vote from peers e485ea0dfa084e81b7c8205293de6527 (127.10.98.1:41323), ecb0141621ca4a1c9d4f8e7f65f76653 (127.10.98.2:42427)
I20260926 12:33:29.178424 12524 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bbcf21d051041c8a0691db6c9723ae2" candidate_uuid: "ab7caa772a3844ea94caf798f45a9a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e485ea0dfa084e81b7c8205293de6527"
I20260926 12:33:29.178532 12524 raft_consensus.cc:3037] T 5bbcf21d051041c8a0691db6c9723ae2 P e485ea0dfa084e81b7c8205293de6527 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:29.179638 12524 raft_consensus.cc:2454] T 5bbcf21d051041c8a0691db6c9723ae2 P e485ea0dfa084e81b7c8205293de6527 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7caa772a3844ea94caf798f45a9a4a in term 1.
I20260926 12:33:29.180080 12664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bbcf21d051041c8a0691db6c9723ae2" candidate_uuid: "ab7caa772a3844ea94caf798f45a9a4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653"
I20260926 12:33:29.180194 12664 raft_consensus.cc:3037] T 5bbcf21d051041c8a0691db6c9723ae2 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:29.180873 12970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:29.176263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:46091 (local address 127.10.98.2:42427)
0926 12:33:29.176383 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:29.176387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:29.176600 (+ 213us) server_negotiation.cc:407] Connection header received
0926 12:33:29.176779 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:29.176782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:29.176842 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:29.176889 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:29.177335 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.178065 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.179555 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.180162 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.180242 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:29.180305 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:29.180679 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 12:33:29.180756 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:33:29.181303 12984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:29.176175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:42427 (local address 127.10.98.1:46091)
0926 12:33:29.176517 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:29.176531 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:29.176665 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:29.177044 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:29.177046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:29.177064 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:29.177220 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:29.177227 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.178213 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.178216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:29.179419 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:29.179429 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.180417 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.180420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:29.180514 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:29.180565 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 12:33:29.181050 (+ 485us) client_negotiation.cc:240] Negotiation successful
0926 12:33:29.181158 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":99,"threads_started":1}
I20260926 12:33:29.181305 12664 raft_consensus.cc:2454] T 5bbcf21d051041c8a0691db6c9723ae2 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab7caa772a3844ea94caf798f45a9a4a in term 1.
I20260926 12:33:29.181598 12735 leader_election.cc:302] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ab7caa772a3844ea94caf798f45a9a4a, ecb0141621ca4a1c9d4f8e7f65f76653; no voters:
I20260926 12:33:29.181867 12664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bbcf21d051041c8a0691db6c9723ae2" candidate_uuid: "e485ea0dfa084e81b7c8205293de6527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" is_pre_election: true
I20260926 12:33:29.182026 12664 raft_consensus.cc:2379] T 5bbcf21d051041c8a0691db6c9723ae2 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e485ea0dfa084e81b7c8205293de6527 in current term 1: Already voted for candidate ab7caa772a3844ea94caf798f45a9a4a in this term.
I20260926 12:33:29.182304 12979 raft_consensus.cc:2781] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:29.182394 12979 raft_consensus.cc:686] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a [term 1 LEADER]: Becoming Leader. State: Replica: ab7caa772a3844ea94caf798f45a9a4a, State: Running, Role: LEADER
I20260926 12:33:29.182538 12979 consensus_queue.cc:237] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } }
I20260926 12:33:29.183316 12965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:29.178186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:60151 (local address 127.10.98.3:33653)
0926 12:33:29.178299 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:29.178303 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:29.178314 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:33:29.178353 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:29.178356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:29.178395 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:29.178466 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:29.178817 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.179567 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.181519 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.182121 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.182191 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:29.182247 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:29.183141 (+ 894us) server_negotiation.cc:299] Negotiation successful
0926 12:33:29.183220 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 12:33:29.183319 12985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:29.177714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:33653 (local address 127.10.98.1:60151)
0926 12:33:29.178024 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:29.178038 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:29.178127 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:29.178562 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:29.178565 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:29.178580 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:29.178703 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:29.178710 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.179704 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.179708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:29.181378 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:29.181388 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.182920 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.182924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:29.183026 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:29.183046 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 12:33:29.183111 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 12:33:29.183174 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":102,"threads_started":1}
I20260926 12:33:29.183737 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bbcf21d051041c8a0691db6c9723ae2" candidate_uuid: "e485ea0dfa084e81b7c8205293de6527" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab7caa772a3844ea94caf798f45a9a4a" is_pre_election: true
I20260926 12:33:29.184235 12366 catalog_manager.cc:5495] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a reported cstate change: term changed from 0 to 1, leader changed from <none> to ab7caa772a3844ea94caf798f45a9a4a (127.10.98.3). New cstate: current_term: 1 leader_uuid: "ab7caa772a3844ea94caf798f45a9a4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab7caa772a3844ea94caf798f45a9a4a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33653 } health_report { overall_health: HEALTHY } } }
I20260926 12:33:29.184599 12458 leader_election.cc:302] T 5bbcf21d051041c8a0691db6c9723ae2 P e485ea0dfa084e81b7c8205293de6527 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e485ea0dfa084e81b7c8205293de6527; no voters: ab7caa772a3844ea94caf798f45a9a4a, ecb0141621ca4a1c9d4f8e7f65f76653
I20260926 12:33:29.184881 12962 raft_consensus.cc:2726] T 5bbcf21d051041c8a0691db6c9723ae2 P e485ea0dfa084e81b7c8205293de6527 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 12:33:29.207985 12965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:29.205028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45160 (local address 127.10.98.3:33653)
0926 12:33:29.205170 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:29.205174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:29.205188 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:33:29.205327 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:29.205331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:29.205400 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:29.205461 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:29.205708 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.206216 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.206997 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:29.207214 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:29.207299 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:29.207540 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:29.207618 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:29.207675 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:29.207793 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 12:33:29.207850 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 12:33:29.216334 12664 raft_consensus.cc:1261] T 5bbcf21d051041c8a0691db6c9723ae2 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 1 FOLLOWER]: Refusing update from remote peer ab7caa772a3844ea94caf798f45a9a4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:29.216325 12524 raft_consensus.cc:1261] T 5bbcf21d051041c8a0691db6c9723ae2 P e485ea0dfa084e81b7c8205293de6527 [term 1 FOLLOWER]: Refusing update from remote peer ab7caa772a3844ea94caf798f45a9a4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:29.216841 12913 consensus_queue.cc:1035] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "ecb0141621ca4a1c9d4f8e7f65f76653" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 42427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:29.217000 12979 consensus_queue.cc:1035] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "e485ea0dfa084e81b7c8205293de6527" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41323 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 12:33:29.233135 12925 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 12:33:30.784983 12366 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:30.785154 12366 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46226:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:30.787453 12366 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6330d03515904304a66a0196d784af0c: Sending DeleteTablet for 3 replicas of tablet 5bbcf21d051041c8a0691db6c9723ae2
I20260926 12:33:30.788039 12504 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bbcf21d051041c8a0691db6c9723ae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:30 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:58766
I20260926 12:33:30.788059 12644 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bbcf21d051041c8a0691db6c9723ae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:30 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:32826
I20260926 12:33:30.788079 12784 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bbcf21d051041c8a0691db6c9723ae2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:30 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:59848
I20260926 12:33:30.788434 13010 tablet_replica.cc:330] T 5bbcf21d051041c8a0691db6c9723ae2 P e485ea0dfa084e81b7c8205293de6527: stopping tablet replica
I20260926 12:33:30.788503 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 12439
I20260926 12:33:30.788399 13009 tablet_replica.cc:330] T 5bbcf21d051041c8a0691db6c9723ae2 P ecb0141621ca4a1c9d4f8e7f65f76653: stopping tablet replica
I20260926 12:33:30.789269 13009 raft_consensus.cc:2229] T 5bbcf21d051041c8a0691db6c9723ae2 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:30.789496 13009 raft_consensus.cc:2258] T 5bbcf21d051041c8a0691db6c9723ae2 P ecb0141621ca4a1c9d4f8e7f65f76653 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:30.789616 13011 tablet_replica.cc:330] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a: stopping tablet replica
I20260926 12:33:30.789748 13011 raft_consensus.cc:2229] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:33:30.789954 13011 raft_consensus.cc:2258] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:30.790227 13009 ts_tablet_manager.cc:1905] T 5bbcf21d051041c8a0691db6c9723ae2 P ecb0141621ca4a1c9d4f8e7f65f76653: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:30.790753 13011 ts_tablet_manager.cc:1905] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:30.793087 13011 ts_tablet_manager.cc:1918] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 12:33:30.793191 13011 log.cc:1196] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/5bbcf21d051041c8a0691db6c9723ae2
I20260926 12:33:30.793322 13009 ts_tablet_manager.cc:1918] T 5bbcf21d051041c8a0691db6c9723ae2 P ecb0141621ca4a1c9d4f8e7f65f76653: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 12:33:30.793397 13009 log.cc:1196] T 5bbcf21d051041c8a0691db6c9723ae2 P ecb0141621ca4a1c9d4f8e7f65f76653: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/5bbcf21d051041c8a0691db6c9723ae2
I20260926 12:33:30.793494 13011 ts_tablet_manager.cc:1939] T 5bbcf21d051041c8a0691db6c9723ae2 P ab7caa772a3844ea94caf798f45a9a4a: Deleting consensus metadata
I20260926 12:33:30.793664 13009 ts_tablet_manager.cc:1939] T 5bbcf21d051041c8a0691db6c9723ae2 P ecb0141621ca4a1c9d4f8e7f65f76653: Deleting consensus metadata
I20260926 12:33:30.794497 12354 catalog_manager.cc:4841] TS ab7caa772a3844ea94caf798f45a9a4a (127.10.98.3:33653): tablet 5bbcf21d051041c8a0691db6c9723ae2 (table test-table [id=2034f845f46b48fc95d5e1a3d02b821b]) successfully deleted
I20260926 12:33:30.794497 12351 catalog_manager.cc:4841] TS ecb0141621ca4a1c9d4f8e7f65f76653 (127.10.98.2:42427): tablet 5bbcf21d051041c8a0691db6c9723ae2 (table test-table [id=2034f845f46b48fc95d5e1a3d02b821b]) successfully deleted
W20260926 12:33:30.795910 12352 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.98.1:41323 (error 108)
I20260926 12:33:30.796087 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 12579
I20260926 12:33:30.801515 13013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:30.797065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:41323 (local address 127.0.0.1:43626)
0926 12:33:30.801332 (+ 4267us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:30.801418 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.1:41323: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4176,"thread_start_us":101,"threads_started":1}
W20260926 12:33:30.801674 12352 catalog_manager.cc:4584] TS e485ea0dfa084e81b7c8205293de6527 (127.10.98.1:41323): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5bbcf21d051041c8a0691db6c9723ae2: Network error: Client connection negotiation failed: client connection to 127.10.98.1:41323: connect: Connection refused (error 111)
I20260926 12:33:30.802323 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 12719
I20260926 12:33:30.808445 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 12335
2026-09-26T12:33:30Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6157 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:33:30 dist-test-slave-t7tk krb5kdc[13016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:33:30 dist-test-slave-t7tk krb5kdc[13016](info): set up 2 sockets
Sep 26 12:33:30 dist-test-slave-t7tk krb5kdc[13016](info): commencing operation
krb5kdc: starting...
W20260926 12:33:31.774663 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:33:31 dist-test-slave-t7tk krb5kdc[13016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:33:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:33:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:33:31.898586 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:46307
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:46307
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:32.008720 13029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:32.009001 13029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:32.009042 13029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:32.013128 13029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:32.013198 13029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:32.013216 13029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:32.013230 13029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:32.013244 13029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:33:32.018666 13029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:46307
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:46307
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:32.019810 13029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:32.020862 13029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:33:32.029773 13029 server_base.cc:1034] running on GCE node
W20260926 12:33:32.030479 13035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:32.030578 13037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:32.031181 13034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:32.031826 13029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:32.033037 13029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:32.034226 13029 hybrid_clock.cc:648] HybridClock initialized: now 1790426012034196 us; error 49 us; skew 500 ppm
Sep 26 12:33:32 dist-test-slave-t7tk krb5kdc[13016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426012, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:32.042614 13029 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:33:32.043694 13029 webserver.cc:460] Webserver started at http://127.10.98.62:39655/ using document root <none> and password file <none>
I20260926 12:33:32.044292 13029 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:32.044364 13029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:32.044557 13029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:32.046697 13029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "02467fc7634047ad8d768156f347409c"
format_stamp: "Formatted at 2026-09-26 12:33:32 on dist-test-slave-t7tk"
server_key: "f09f61014ec3c0394c7b89723e952330"
server_key_iv: "58aaba1a95678eaaa5dac8576fcd8569"
server_key_version: "encryptionkey@0"
I20260926 12:33:32.047245 13029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "02467fc7634047ad8d768156f347409c"
format_stamp: "Formatted at 2026-09-26 12:33:32 on dist-test-slave-t7tk"
server_key: "f09f61014ec3c0394c7b89723e952330"
server_key_iv: "58aaba1a95678eaaa5dac8576fcd8569"
server_key_version: "encryptionkey@0"
I20260926 12:33:32.051101 13029 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:33:32.053732 13044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:32.055099 13029 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:33:32.055229 13029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "02467fc7634047ad8d768156f347409c"
format_stamp: "Formatted at 2026-09-26 12:33:32 on dist-test-slave-t7tk"
server_key: "f09f61014ec3c0394c7b89723e952330"
server_key_iv: "58aaba1a95678eaaa5dac8576fcd8569"
server_key_version: "encryptionkey@0"
I20260926 12:33:32.055332 13029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:32.078464 13029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:32.081775 13029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:32.081954 13029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:32.089287 13029 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:46307
I20260926 12:33:32.089314 13096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:46307 every 8 connection(s)
I20260926 12:33:32.090420 13029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:33:32.093777 13097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:32.095068 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 13029
I20260926 12:33:32.095170 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:33:32.095434 10632 external_mini_cluster.cc:1428] Setting key dab54b2b64e9ea136651a35814bf091a
I20260926 12:33:32.100466 13097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c: Bootstrap starting.
I20260926 12:33:32.102972 13097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:32.103853 13097 log.cc:824] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:33:32 dist-test-slave-t7tk krb5kdc[13016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:32.106487 13097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c: No bootstrap required, opened a new log
I20260926 12:33:32.110139 13097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02467fc7634047ad8d768156f347409c" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 46307 } }
I20260926 12:33:32.110394 13097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:32.110443 13097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02467fc7634047ad8d768156f347409c, State: Initialized, Role: FOLLOWER
I20260926 12:33:32.110936 13097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02467fc7634047ad8d768156f347409c" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 46307 } }
I20260926 12:33:32.111066 13097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:33:32.111044 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:32.097197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.10.98.62:46307)
0926 12:33:32.097762 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:32.097781 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:32.097833 (+ 52us) server_negotiation.cc:407] Connection header received
0926 12:33:32.098600 (+ 767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:32.098623 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:32.099132 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:32.099553 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:32.100411 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:32.101543 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:32.102432 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:32.102950 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:32.103108 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:32.106046 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:32.106053 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:32.106071 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:32.106106 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:32.107950 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:32.108408 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:32.108414 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:32.108420 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:32.108549 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:32.108818 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:32.108822 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:32.108823 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:32.109308 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:32.109531 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:32.109797 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 12:33:32.110101 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":151,"threads_started":1}
I20260926 12:33:32.111150 13097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:33:32.111239 13097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:32.112272 13097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02467fc7634047ad8d768156f347409c" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 46307 } }
I20260926 12:33:32.112648 13097 leader_election.cc:302] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 02467fc7634047ad8d768156f347409c; no voters:
I20260926 12:33:32.113034 13097 leader_election.cc:288] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:33:32.113191 13102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:32.113405 13102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c [term 1 LEADER]: Becoming Leader. State: Replica: 02467fc7634047ad8d768156f347409c, State: Running, Role: LEADER
I20260926 12:33:32.113755 13102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02467fc7634047ad8d768156f347409c" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 46307 } }
I20260926 12:33:32.114333 13097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:32.115372 13104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "02467fc7634047ad8d768156f347409c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02467fc7634047ad8d768156f347409c" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 46307 } } }
I20260926 12:33:32.115502 13104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:32.115887 13111 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:33:32.115845 13103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02467fc7634047ad8d768156f347409c. Latest consensus state: current_term: 1 leader_uuid: "02467fc7634047ad8d768156f347409c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "02467fc7634047ad8d768156f347409c" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 46307 } } }
I20260926 12:33:32.116016 13103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:32.118858 13111 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:33:32.124553 13111 catalog_manager.cc:1347] Generated new cluster ID: bba5ad1217da4f17b1baa1015f1ee5e0
I20260926 12:33:32.124686 13111 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:33:32.135582 13111 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:33:32.136926 13111 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:33:32.151319 13111 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 02467fc7634047ad8d768156f347409c: Generated new TSK 0
I20260926 12:33:32.152225 13111 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
I20260926 12:33:32.185554 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:46307
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:32.299664 13123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:32.299937 13123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:32.299994 13123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:32.304397 13123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:32.304481 13123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:32.304566 13123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:33:32.309911 13123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13123
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:32.311223 13123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:32.312400 13123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:32.321976 13128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:32.321969 13131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:32.321978 13129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:32.322427 13123 server_base.cc:1034] running on GCE node
I20260926 12:33:32.322820 13123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:32.323446 13123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:32.324613 13123 hybrid_clock.cc:648] HybridClock initialized: now 1790426012324598 us; error 35 us; skew 500 ppm
Sep 26 12:33:32 dist-test-slave-t7tk krb5kdc[13016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426012, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:32.327293 13123 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:33:32.328154 13123 webserver.cc:460] Webserver started at http://127.10.98.1:45207/ using document root <none> and password file <none>
I20260926 12:33:32.328685 13123 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:32.328768 13123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:32.329025 13123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:32.331087 13123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "7c11ef5595534dba9ecd2c336de84d85"
format_stamp: "Formatted at 2026-09-26 12:33:32 on dist-test-slave-t7tk"
server_key: "1fe7c7268f66f0b9add440f9a2336dbd"
server_key_iv: "e1dd0f6e7065e2f463a8bd91a8e273d6"
server_key_version: "encryptionkey@0"
I20260926 12:33:32.331640 13123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "7c11ef5595534dba9ecd2c336de84d85"
format_stamp: "Formatted at 2026-09-26 12:33:32 on dist-test-slave-t7tk"
server_key: "1fe7c7268f66f0b9add440f9a2336dbd"
server_key_iv: "e1dd0f6e7065e2f463a8bd91a8e273d6"
server_key_version: "encryptionkey@0"
I20260926 12:33:32.335239 13123 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 12:33:32.337598 13138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:32.338828 13123 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:33:32.338949 13123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "7c11ef5595534dba9ecd2c336de84d85"
format_stamp: "Formatted at 2026-09-26 12:33:32 on dist-test-slave-t7tk"
server_key: "1fe7c7268f66f0b9add440f9a2336dbd"
server_key_iv: "e1dd0f6e7065e2f463a8bd91a8e273d6"
server_key_version: "encryptionkey@0"
I20260926 12:33:32.339047 13123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:32.355511 13123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:32.358687 13123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:32.358879 13123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:32.359476 13123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:32.360291 13123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:32.360347 13123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:32.360402 13123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:32.360428 13123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:32.368957 13123 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:36879
I20260926 12:33:32.369014 13251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:36879 every 8 connection(s)
I20260926 12:33:32.369863 13123 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:33:32.371999 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 13123
I20260926 12:33:32.372097 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:33:32.372339 10632 external_mini_cluster.cc:1428] Setting key 35cded0ca54cda9387fe6ad388194797
Sep 26 12:33:32 dist-test-slave-t7tk krb5kdc[13016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426012, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.2@KRBTEST.COM" created.
I20260926 12:33:32.384795 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:32.371793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:46959 (local address 127.10.98.62:46307)
0926 12:33:32.371967 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:32.371972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:32.372950 (+ 978us) server_negotiation.cc:407] Connection header received
0926 12:33:32.374554 (+ 1604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:32.374557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:32.374600 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:32.374679 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:32.376123 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:32.376651 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:32.378011 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:32.378363 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:32.378481 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:32.381120 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:32.381125 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:32.381128 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:32.381157 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:32.382624 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:32.383162 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:32.383165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:32.383166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:32.383250 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:32.383606 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:32.383609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:32.383610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:32.383855 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:32.383940 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:32.384545 (+ 605us) server_negotiation.cc:299] Negotiation successful
0926 12:33:32.384670 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 12:33:32.385934 13254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:32.372233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:46307 (local address 127.10.98.1:46959)
0926 12:33:32.372748 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:32.372792 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:32.374300 (+ 1508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:32.374881 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:32.374890 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:32.375362 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:32.375984 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:32.375999 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:32.376825 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:32.376829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:32.377886 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:32.377899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:32.378575 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:32.378578 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:32.378727 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:32.379334 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:32.379351 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:32.380906 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:32.382735 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:32.382740 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:32.382753 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:32.383062 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:32.383330 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:32.383332 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:32.383333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:32.383499 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:32.383975 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:32.383980 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:32.384263 (+ 283us) client_negotiation.cc:769] Sending connection context
0926 12:33:32.384486 (+ 223us) client_negotiation.cc:240] Negotiation successful
0926 12:33:32.384828 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":142,"threads_started":1}
I20260926 12:33:32.387210 13252 heartbeater.cc:347] Connected to a master server at 127.10.98.62:46307
I20260926 12:33:32.387465 13252 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:32.388052 13252 heartbeater.cc:511] Master 127.10.98.62:46307 requested a full tablet report, sending...
I20260926 12:33:32.389945 13061 ts_manager.cc:194] Registered new tserver with Master: 7c11ef5595534dba9ecd2c336de84d85 (127.10.98.1:36879)
I20260926 12:33:32.391530 13061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:46959
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
I20260926 12:33:32.402168 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:46307
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:32.512526 13257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:32.512809 13257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:32.512852 13257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:32.517005 13257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:32.517074 13257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:32.517166 13257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:33:32.522678 13257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:32.523736 13257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:32.524770 13257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:33:32.534190 13257 server_base.cc:1034] running on GCE node
W20260926 12:33:32.534235 13262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:32.534164 13263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:32.534317 13265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:32.534821 13257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:32.535522 13257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:32.536695 13257 hybrid_clock.cc:648] HybridClock initialized: now 1790426012536678 us; error 35 us; skew 500 ppm
Sep 26 12:33:32 dist-test-slave-t7tk krb5kdc[13016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426012, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:32.539362 13257 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:33:32.540179 13257 webserver.cc:460] Webserver started at http://127.10.98.2:33897/ using document root <none> and password file <none>
I20260926 12:33:32.540711 13257 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:32.540769 13257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:32.540931 13257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:32.543037 13257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "3d872ebf41fa46d093d948d39a220b0a"
format_stamp: "Formatted at 2026-09-26 12:33:32 on dist-test-slave-t7tk"
server_key: "0593094c5cddfc7fb1a465cf0137c2c0"
server_key_iv: "eea34468c39d975968a5c66966dcd328"
server_key_version: "encryptionkey@0"
I20260926 12:33:32.543576 13257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "3d872ebf41fa46d093d948d39a220b0a"
format_stamp: "Formatted at 2026-09-26 12:33:32 on dist-test-slave-t7tk"
server_key: "0593094c5cddfc7fb1a465cf0137c2c0"
server_key_iv: "eea34468c39d975968a5c66966dcd328"
server_key_version: "encryptionkey@0"
I20260926 12:33:32.547318 13257 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:33:32.549731 13272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:32.551095 13257 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:33:32.551205 13257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "3d872ebf41fa46d093d948d39a220b0a"
format_stamp: "Formatted at 2026-09-26 12:33:32 on dist-test-slave-t7tk"
server_key: "0593094c5cddfc7fb1a465cf0137c2c0"
server_key_iv: "eea34468c39d975968a5c66966dcd328"
server_key_version: "encryptionkey@0"
I20260926 12:33:32.551299 13257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:32.568676 13257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:32.571898 13257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:32.572048 13257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:32.572573 13257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:32.573362 13257 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:32.573406 13257 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:32.573443 13257 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:32.573463 13257 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:32.581903 13257 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:33357
I20260926 12:33:32.581986 13385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:33357 every 8 connection(s)
I20260926 12:33:32.582795 13257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:33:32.588519 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 13257
I20260926 12:33:32.588627 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:33:32.588876 10632 external_mini_cluster.cc:1428] Setting key 2fb9236676f7d6559b8e4fe52b1de8ea
Sep 26 12:33:32 dist-test-slave-t7tk krb5kdc[13016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426012, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:32.596706 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:32.584628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:34945 (local address 127.10.98.62:46307)
0926 12:33:32.584749 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:32.584753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:32.585645 (+ 892us) server_negotiation.cc:407] Connection header received
0926 12:33:32.586918 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:32.586921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:32.587012 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:32.587088 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:32.588268 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:32.588825 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:32.590176 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:32.590400 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:32.590489 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:32.593006 (+ 2517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:32.593009 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:32.593012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:32.593037 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:32.594552 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:32.595203 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:32.595207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:32.595209 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:32.595300 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:32.595639 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:32.595643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:32.595645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:32.595835 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:32.595924 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:32.596419 (+ 495us) server_negotiation.cc:299] Negotiation successful
0926 12:33:32.596547 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.10.98.3@KRBTEST.COM; defaulting to no policy
I20260926 12:33:32.597589 13388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:32.585029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:46307 (local address 127.10.98.2:34945)
0926 12:33:32.585504 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:32.585545 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:32.586711 (+ 1166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:32.587273 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:32.587280 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:32.587635 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:32.588110 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:32.588122 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:32.588942 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:32.588945 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:32.590068 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:32.590078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:32.590536 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:32.590540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:32.590707 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:32.591261 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:32.591277 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:32.592826 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:32.594664 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:32.594670 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:32.594685 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:32.595087 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:32.595386 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:32.595389 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:32.595391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:32.595526 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:32.595938 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:32.595945 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:32.596188 (+ 243us) client_negotiation.cc:769] Sending connection context
0926 12:33:32.596358 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 12:33:32.596687 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
I20260926 12:33:32.598773 13386 heartbeater.cc:347] Connected to a master server at 127.10.98.62:46307
I20260926 12:33:32.598999 13386 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:32.599509 13386 heartbeater.cc:511] Master 127.10.98.62:46307 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.3@KRBTEST.COM" created.
I20260926 12:33:32.600636 13061 ts_manager.cc:194] Registered new tserver with Master: 3d872ebf41fa46d093d948d39a220b0a (127.10.98.2:33357)
I20260926 12:33:32.601409 13061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:34945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
I20260926 12:33:32.617239 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:46307
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:46151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:32.724807 13391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:32.725076 13391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:32.725155 13391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:32.729161 13391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:32.729220 13391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:32.729298 13391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:33:32.734565 13391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:46151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:46307
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:32.735574 13391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:32.736572 13391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:32.746107 13396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:32.746148 13399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:32.746181 13397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:32.746327 13391 server_base.cc:1034] running on GCE node
I20260926 12:33:32.746781 13391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:32.747394 13391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:32.748556 13391 hybrid_clock.cc:648] HybridClock initialized: now 1790426012748533 us; error 42 us; skew 500 ppm
Sep 26 12:33:32 dist-test-slave-t7tk krb5kdc[13016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426012, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:32.751252 13391 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:33:32.752112 13391 webserver.cc:460] Webserver started at http://127.10.98.3:33465/ using document root <none> and password file <none>
I20260926 12:33:32.752852 13391 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:32.752952 13391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:32.753250 13391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:32.755415 13391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "b9f9dbd577324c69b719caadde01577d"
format_stamp: "Formatted at 2026-09-26 12:33:32 on dist-test-slave-t7tk"
server_key: "e23442e02672a1971a41d3dc47b53222"
server_key_iv: "e8162444fe4c57b5b928313bba9f46b4"
server_key_version: "encryptionkey@0"
I20260926 12:33:32.755980 13391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "b9f9dbd577324c69b719caadde01577d"
format_stamp: "Formatted at 2026-09-26 12:33:32 on dist-test-slave-t7tk"
server_key: "e23442e02672a1971a41d3dc47b53222"
server_key_iv: "e8162444fe4c57b5b928313bba9f46b4"
server_key_version: "encryptionkey@0"
I20260926 12:33:32.760183 13391 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 12:33:32.762843 13406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:32.764101 13391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:33:32.764199 13391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "b9f9dbd577324c69b719caadde01577d"
format_stamp: "Formatted at 2026-09-26 12:33:32 on dist-test-slave-t7tk"
server_key: "e23442e02672a1971a41d3dc47b53222"
server_key_iv: "e8162444fe4c57b5b928313bba9f46b4"
server_key_version: "encryptionkey@0"
I20260926 12:33:32.764287 13391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:32.808686 13391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:32.811841 13391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:32.811987 13391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:32.812536 13391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:32.813378 13391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:32.813426 13391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:32.813465 13391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:32.813479 13391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:32.821964 13391 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:35545
I20260926 12:33:32.822026 13519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:35545 every 8 connection(s)
I20260926 12:33:32.822849 13391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:33:32.823449 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 13391
I20260926 12:33:32.823554 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:33:32.823834 10632 external_mini_cluster.cc:1428] Setting key c81e68ca0c588bbd306bf9f66d9f1808
Sep 26 12:33:32 dist-test-slave-t7tk krb5kdc[13016](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426012, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:32.835898 13100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:32.824630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:33259 (local address 127.10.98.62:46307)
0926 12:33:32.824747 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:32.824751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:32.825882 (+ 1131us) server_negotiation.cc:407] Connection header received
0926 12:33:32.827159 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:32.827161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:32.827203 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:32.827273 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:32.828372 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:32.828950 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:32.830038 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:32.830279 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:32.830361 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:32.832601 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:32.832604 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:32.832606 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:32.832629 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:32.834025 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:32.834482 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:32.834486 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:32.834487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:32.834559 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:32.834879 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:32.834882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:32.834883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:32.835070 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:32.835153 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:32.835587 (+ 434us) server_negotiation.cc:299] Negotiation successful
0926 12:33:32.835684 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 12:33:32.836735 13522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:32.825180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:46307 (local address 127.10.98.3:33259)
0926 12:33:32.825722 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:32.825778 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:32.826970 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:32.827425 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:32.827432 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:32.827787 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:32.828252 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:32.828264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:32.829040 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:32.829042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:32.829912 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:32.829920 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:32.830396 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:32.830398 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:32.830548 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:32.831051 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:32.831066 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:32.832446 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:32.834123 (+ 1677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:32.834127 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:32.834139 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:32.834397 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:32.834627 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:32.834629 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:32.834630 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:32.834749 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:32.835167 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:32.835173 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:32.835397 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 12:33:32.835552 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 12:33:32.835799 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":133,"threads_started":1}
I20260926 12:33:32.837833 13520 heartbeater.cc:347] Connected to a master server at 127.10.98.62:46307
I20260926 12:33:32.838052 13520 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:32.838493 13520 heartbeater.cc:511] Master 127.10.98.62:46307 requested a full tablet report, sending...
I20260926 12:33:32.839468 13061 ts_manager.cc:194] Registered new tserver with Master: b9f9dbd577324c69b719caadde01577d (127.10.98.3:35545)
I20260926 12:33:32.839998 13061 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:33259
I20260926 12:33:32.849712 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 12:33:32.861905 13100 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:32.860456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56090 (local address 127.10.98.62:46307)
0926 12:33:32.860611 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:32.860615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:32.860722 (+ 107us) server_negotiation.cc:407] Connection header received
0926 12:33:32.860842 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:32.860844 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:32.860886 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:32.860954 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:32.861768 (+ 814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56090: BlockingRecv error: recv got EOF from 127.0.0.1:56090 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 12:33:32.862948 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 13123
I20260926 12:33:32.868012 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 13257
I20260926 12:33:32.872356 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 13391
I20260926 12:33:32.876948 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 13029
2026-09-26T12:33:32Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2067 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:33:32 dist-test-slave-t7tk krb5kdc[13532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:33:32 dist-test-slave-t7tk krb5kdc[13532](info): set up 2 sockets
Sep 26 12:33:32 dist-test-slave-t7tk krb5kdc[13532](info): commencing operation
krb5kdc: starting...
W20260926 12:33:33.842816 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:33:33 dist-test-slave-t7tk krb5kdc[13532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:33:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:33:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:33:33.964547 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:45989
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:36809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:45989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:34.073159 13545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:34.073439 13545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:34.073480 13545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:34.077554 13545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:34.077618 13545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:34.077636 13545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:34.077652 13545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:34.077665 13545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:33:34.083233 13545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:36809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:45989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:45989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:34.084424 13545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:34.085638 13545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:34.094578 13551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:34.094693 13550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:34.094739 13553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:34.094602 13545 server_base.cc:1034] running on GCE node
I20260926 12:33:34.095575 13545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:34.096782 13545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:34.097951 13545 hybrid_clock.cc:648] HybridClock initialized: now 1790426014097910 us; error 58 us; skew 500 ppm
Sep 26 12:33:34 dist-test-slave-t7tk krb5kdc[13532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426014, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:34.100927 13545 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:33:34.101933 13545 webserver.cc:460] Webserver started at http://127.10.98.62:34073/ using document root <none> and password file <none>
I20260926 12:33:34.102475 13545 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:34.102542 13545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:34.102731 13545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:34.104789 13545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "7a7286ed8d0646e681d94f4ae921a24e"
format_stamp: "Formatted at 2026-09-26 12:33:34 on dist-test-slave-t7tk"
server_key: "be0e26b7361aa589b8772cb05f111f1b"
server_key_iv: "fc68e1e0cf820c42021a1ff051470c78"
server_key_version: "encryptionkey@0"
I20260926 12:33:34.105360 13545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "7a7286ed8d0646e681d94f4ae921a24e"
format_stamp: "Formatted at 2026-09-26 12:33:34 on dist-test-slave-t7tk"
server_key: "be0e26b7361aa589b8772cb05f111f1b"
server_key_iv: "fc68e1e0cf820c42021a1ff051470c78"
server_key_version: "encryptionkey@0"
I20260926 12:33:34.109012 13545 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:33:34.111493 13560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:34.112747 13545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:33:34.112859 13545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "7a7286ed8d0646e681d94f4ae921a24e"
format_stamp: "Formatted at 2026-09-26 12:33:34 on dist-test-slave-t7tk"
server_key: "be0e26b7361aa589b8772cb05f111f1b"
server_key_iv: "fc68e1e0cf820c42021a1ff051470c78"
server_key_version: "encryptionkey@0"
I20260926 12:33:34.112996 13545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:34.130062 13545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:34.133244 13545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:34.133425 13545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:34.140194 13545 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:45989
I20260926 12:33:34.140535 13612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:45989 every 8 connection(s)
I20260926 12:33:34.141235 13545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:33:34.144122 13613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:34.150202 13613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e: Bootstrap starting.
I20260926 12:33:34.150592 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 13545
I20260926 12:33:34.150686 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:33:34.150919 10632 external_mini_cluster.cc:1428] Setting key 94240c9d1c308fa3925d069a753b3531
I20260926 12:33:34.152776 13613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:34.153688 13613 log.cc:824] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:34.155596 13613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e: No bootstrap required, opened a new log
I20260926 12:33:34.159026 13613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7286ed8d0646e681d94f4ae921a24e" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45989 } }
I20260926 12:33:34.159261 13613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:34.159317 13613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a7286ed8d0646e681d94f4ae921a24e, State: Initialized, Role: FOLLOWER
Sep 26 12:33:34 dist-test-slave-t7tk krb5kdc[13532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:34.159735 13613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7286ed8d0646e681d94f4ae921a24e" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45989 } }
I20260926 12:33:34.159898 13613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:33:34.159970 13613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:33:34.160069 13613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:34.160985 13613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7286ed8d0646e681d94f4ae921a24e" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45989 } }
I20260926 12:33:34.161350 13613 leader_election.cc:302] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a7286ed8d0646e681d94f4ae921a24e; no voters:
I20260926 12:33:34.161664 13613 leader_election.cc:288] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:33:34.161778 13618 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:34.161983 13618 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e [term 1 LEADER]: Becoming Leader. State: Replica: 7a7286ed8d0646e681d94f4ae921a24e, State: Running, Role: LEADER
I20260926 12:33:34.162392 13618 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7286ed8d0646e681d94f4ae921a24e" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45989 } }
I20260926 12:33:34.162955 13613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:34.164103 13619 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a7286ed8d0646e681d94f4ae921a24e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7286ed8d0646e681d94f4ae921a24e" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45989 } } }
I20260926 12:33:34.164227 13619 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:34.164659 13620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a7286ed8d0646e681d94f4ae921a24e. Latest consensus state: current_term: 1 leader_uuid: "7a7286ed8d0646e681d94f4ae921a24e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a7286ed8d0646e681d94f4ae921a24e" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45989 } } }
I20260926 12:33:34.164821 13620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:34.166136 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:34.152345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51184 (local address 127.10.98.62:45989)
0926 12:33:34.152786 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:34.152805 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:34.152849 (+ 44us) server_negotiation.cc:407] Connection header received
0926 12:33:34.153719 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:34.153743 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:34.154168 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:34.154499 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:34.155282 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:34.156303 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:34.157158 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:34.157647 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:34.157790 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:34.160326 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:34.160333 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:34.160347 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:34.160377 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:34.162261 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:34.162798 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:34.162804 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:34.162810 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:34.162958 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:34.163268 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:34.163272 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:34.163273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:34.163822 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:34.164018 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:34.164928 (+ 910us) server_negotiation.cc:299] Negotiation successful
0926 12:33:34.165273 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":110,"threads_started":1}
W20260926 12:33:34.168854 13633 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:33:34.168970 13633 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:33:34.169003 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:33:34.169037 13627 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:33:34.169847 13627 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:33:34.175700 13627 catalog_manager.cc:1347] Generated new cluster ID: a1a94ad6655c49caafca95ea7ab124fd
I20260926 12:33:34.175767 13627 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:33:34.185859 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:34.181205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51194 (local address 127.10.98.62:45989)
0926 12:33:34.181333 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:34.181338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:34.181559 (+ 221us) server_negotiation.cc:407] Connection header received
0926 12:33:34.181676 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:34.181679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:34.181722 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:34.181788 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:34.182301 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:34.182827 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:34.183730 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:34.184033 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:34.184123 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:34.184900 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:34.184904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:34.184907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:33:34.185419 (+ 512us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:34.185453 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:33:34.185470 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:34.185550 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:34.185650 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 12:33:34.185717 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:33:34.202390 13627 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:33:34.203694 13627 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:33:34.210835 13627 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a7286ed8d0646e681d94f4ae921a24e: Generated new TSK 0
I20260926 12:33:34.211603 13627 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:33:34.216979 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 13545
2026-09-26T12:33:34Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1337 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:33:34 dist-test-slave-t7tk krb5kdc[13646](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:33:34 dist-test-slave-t7tk krb5kdc[13646](info): set up 2 sockets
Sep 26 12:33:34 dist-test-slave-t7tk krb5kdc[13646](info): commencing operation
krb5kdc: starting...
W20260926 12:33:35.170603 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:33:35 dist-test-slave-t7tk krb5kdc[13646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:33:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:33:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:33:35.290627 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:36501
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:41961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:36501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:35.400238 13659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:35.400542 13659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:35.400601 13659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:35.404659 13659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:35.404726 13659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:35.404752 13659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:35.404773 13659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:35.404794 13659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:33:35.410229 13659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:41961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:36501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:36501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:35.411298 13659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:35.412323 13659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:35.420863 13665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:35.420836 13664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:35.420833 13667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:35.421309 13659 server_base.cc:1034] running on GCE node
I20260926 12:33:35.421866 13659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:35.422940 13659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:35.424109 13659 hybrid_clock.cc:648] HybridClock initialized: now 1790426015424084 us; error 39 us; skew 500 ppm
Sep 26 12:33:35 dist-test-slave-t7tk krb5kdc[13646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426015, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:35.426746 13659 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:33:35.427670 13659 webserver.cc:460] Webserver started at http://127.10.98.62:44477/ using document root <none> and password file <none>
I20260926 12:33:35.428226 13659 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:35.428295 13659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:35.428521 13659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:35.430637 13659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "c69170a896d3489c9cd1319be3839807"
format_stamp: "Formatted at 2026-09-26 12:33:35 on dist-test-slave-t7tk"
server_key: "eca06cb59ac3dff643e8f6df9680a556"
server_key_iv: "0d51660fe823e9b538800e1998faa34f"
server_key_version: "encryptionkey@0"
I20260926 12:33:35.431170 13659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "c69170a896d3489c9cd1319be3839807"
format_stamp: "Formatted at 2026-09-26 12:33:35 on dist-test-slave-t7tk"
server_key: "eca06cb59ac3dff643e8f6df9680a556"
server_key_iv: "0d51660fe823e9b538800e1998faa34f"
server_key_version: "encryptionkey@0"
I20260926 12:33:35.434811 13659 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:33:35.437266 13674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:35.438586 13659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:33:35.438712 13659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "c69170a896d3489c9cd1319be3839807"
format_stamp: "Formatted at 2026-09-26 12:33:35 on dist-test-slave-t7tk"
server_key: "eca06cb59ac3dff643e8f6df9680a556"
server_key_iv: "0d51660fe823e9b538800e1998faa34f"
server_key_version: "encryptionkey@0"
I20260926 12:33:35.438809 13659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:35.466984 13659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:35.470252 13659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:35.470448 13659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:35.480170 13659 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:36501
I20260926 12:33:35.480489 13726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:36501 every 8 connection(s)
I20260926 12:33:35.481464 13659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:33:35.484396 13727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:35.487085 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 13659
I20260926 12:33:35.487190 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:33:35.487478 10632 external_mini_cluster.cc:1428] Setting key c68a469fb0e9f5dc69c2dcf5bcaa8f7c
I20260926 12:33:35.490648 13727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807: Bootstrap starting.
I20260926 12:33:35.493384 13727 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:35.494161 13727 log.cc:824] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:35.496275 13727 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807: No bootstrap required, opened a new log
Sep 26 12:33:35 dist-test-slave-t7tk krb5kdc[13646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426015, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:35.500124 13727 raft_consensus.cc:348] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69170a896d3489c9cd1319be3839807" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 36501 } }
I20260926 12:33:35.500411 13727 raft_consensus.cc:374] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:35.500476 13727 raft_consensus.cc:729] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c69170a896d3489c9cd1319be3839807, State: Initialized, Role: FOLLOWER
I20260926 12:33:35.500941 13727 consensus_queue.cc:260] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69170a896d3489c9cd1319be3839807" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 36501 } }
I20260926 12:33:35.501091 13727 raft_consensus.cc:388] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:33:35.501209 13727 raft_consensus.cc:482] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:33:35.501308 13727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:35.502457 13727 raft_consensus.cc:504] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69170a896d3489c9cd1319be3839807" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 36501 } }
I20260926 12:33:35.502890 13727 leader_election.cc:302] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c69170a896d3489c9cd1319be3839807; no voters:
I20260926 12:33:35.503223 13727 leader_election.cc:288] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:33:35.503356 13732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:35.503587 13732 raft_consensus.cc:686] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807 [term 1 LEADER]: Becoming Leader. State: Replica: c69170a896d3489c9cd1319be3839807, State: Running, Role: LEADER
I20260926 12:33:35.503832 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:35.489042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44014 (local address 127.10.98.62:36501)
0926 12:33:35.489544 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:35.489564 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:35.489609 (+ 45us) server_negotiation.cc:407] Connection header received
0926 12:33:35.490419 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:35.490443 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:35.490865 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:35.491232 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:35.491942 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:35.493472 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:35.494426 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:35.494982 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:35.495134 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:35.497845 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:35.497851 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:35.497869 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:35.497902 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:35.500535 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:35.501052 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:35.501058 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:35.501064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:35.501249 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:35.501527 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:35.501530 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:35.501532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:35.502034 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:35.502230 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:35.502495 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 12:33:35.502805 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":148,"threads_started":1}
I20260926 12:33:35.504046 13732 consensus_queue.cc:237] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69170a896d3489c9cd1319be3839807" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 36501 } }
I20260926 12:33:35.504519 13727 sys_catalog.cc:564] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:35.505187 13732 sys_catalog.cc:455] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c69170a896d3489c9cd1319be3839807. Latest consensus state: current_term: 1 leader_uuid: "c69170a896d3489c9cd1319be3839807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69170a896d3489c9cd1319be3839807" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 36501 } } }
I20260926 12:33:35.505375 13732 sys_catalog.cc:458] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:35.505614 13734 sys_catalog.cc:455] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c69170a896d3489c9cd1319be3839807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c69170a896d3489c9cd1319be3839807" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 36501 } } }
I20260926 12:33:35.505709 13734 sys_catalog.cc:458] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:35.506644 13741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:33:35.509661 13741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:33:35.514955 13741 catalog_manager.cc:1347] Generated new cluster ID: 06fe9a35375e418bb2803ae2746abcb5
I20260926 12:33:35.515024 13741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:33:35.530081 13741 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:33:35.531070 13741 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:33:35.535355 13741 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c69170a896d3489c9cd1319be3839807: Generated new TSK 0
I20260926 12:33:35.536042 13741 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
I20260926 12:33:35.573472 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:36501
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:41961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:35.695084 13753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:35.695366 13753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:35.695436 13753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:35.699651 13753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:35.699726 13753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:35.699816 13753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:33:35.705451 13753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:41961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:36501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:35.706543 13753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:35.707583 13753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:35.717250 13759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:35.717272 13758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:35.717547 13753 server_base.cc:1034] running on GCE node
W20260926 12:33:35.717557 13761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:35.718048 13753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:35.718662 13753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:35.719851 13753 hybrid_clock.cc:648] HybridClock initialized: now 1790426015719794 us; error 69 us; skew 500 ppm
Sep 26 12:33:35 dist-test-slave-t7tk krb5kdc[13646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426015, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:35.722666 13753 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:33:35.723513 13753 webserver.cc:460] Webserver started at http://127.10.98.1:45835/ using document root <none> and password file <none>
I20260926 12:33:35.724031 13753 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:35.724087 13753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:35.724257 13753 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:35.726291 13753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "3a009ef4a8f34f1f84553bd70090f022"
format_stamp: "Formatted at 2026-09-26 12:33:35 on dist-test-slave-t7tk"
server_key: "56bce3b459d5aea9ce2275cc85042d7e"
server_key_iv: "e835e648593f717b0ca63454c1640d5b"
server_key_version: "encryptionkey@0"
I20260926 12:33:35.726816 13753 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "3a009ef4a8f34f1f84553bd70090f022"
format_stamp: "Formatted at 2026-09-26 12:33:35 on dist-test-slave-t7tk"
server_key: "56bce3b459d5aea9ce2275cc85042d7e"
server_key_iv: "e835e648593f717b0ca63454c1640d5b"
server_key_version: "encryptionkey@0"
I20260926 12:33:35.730829 13753 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:33:35.733268 13768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:35.735183 13753 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:33:35.735311 13753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "3a009ef4a8f34f1f84553bd70090f022"
format_stamp: "Formatted at 2026-09-26 12:33:35 on dist-test-slave-t7tk"
server_key: "56bce3b459d5aea9ce2275cc85042d7e"
server_key_iv: "e835e648593f717b0ca63454c1640d5b"
server_key_version: "encryptionkey@0"
I20260926 12:33:35.735438 13753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:35.751932 13753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:35.756012 13753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:35.756213 13753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:35.756879 13753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:35.757915 13753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:35.757970 13753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:35.758031 13753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:35.758054 13753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:35.768208 13753 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:35553
I20260926 12:33:35.768246 13881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:35553 every 8 connection(s)
I20260926 12:33:35.769243 13753 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:33:35.770450 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 13753
I20260926 12:33:35.770555 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:33:35.770840 10632 external_mini_cluster.cc:1428] Setting key 7c96c99e73ff8483e4085fe6af2e0754
WARNING: no policy specified for kudu/127.10.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.2@KRBTEST.COM" created.
Sep 26 12:33:35 dist-test-slave-t7tk krb5kdc[13646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426015, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:35.788120 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:35.771427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:34405 (local address 127.10.98.62:36501)
0926 12:33:35.771657 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:35.771661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:35.772690 (+ 1029us) server_negotiation.cc:407] Connection header received
0926 12:33:35.774608 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:35.774611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:35.774658 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:35.774727 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:35.776420 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:35.777330 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:35.778522 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:35.778879 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:35.778997 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:35.784379 (+ 5382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:35.784382 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:35.784385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:35.784414 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:35.785944 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:35.786485 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:35.786489 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:35.786491 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:35.786589 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:35.786952 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:35.786955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:35.786957 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:35.787175 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:35.787263 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:35.787818 (+ 555us) server_negotiation.cc:299] Negotiation successful
0926 12:33:35.787951 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260926 12:33:35.789037 13884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:35.771933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:36501 (local address 127.10.98.1:34405)
0926 12:33:35.772517 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:35.772583 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:35.774356 (+ 1773us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:35.774943 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:35.774953 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:35.775473 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:35.776230 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:35.776251 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:35.777459 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:35.777461 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:35.778387 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:35.778396 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:35.779074 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:35.779077 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:35.779237 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:35.779850 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:35.779872 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:35.784154 (+ 4282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:35.786067 (+ 1913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:35.786074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:35.786094 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:35.786379 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:35.786686 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:35.786689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:35.786691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:35.786847 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:35.787279 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:35.787287 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:35.787577 (+ 290us) client_negotiation.cc:769] Sending connection context
0926 12:33:35.787759 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 12:33:35.788141 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":153,"threads_started":1}
I20260926 12:33:35.790575 13882 heartbeater.cc:347] Connected to a master server at 127.10.98.62:36501
I20260926 12:33:35.790823 13882 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:35.791339 13882 heartbeater.cc:511] Master 127.10.98.62:36501 requested a full tablet report, sending...
I20260926 12:33:35.793201 13690 ts_manager.cc:194] Registered new tserver with Master: 3a009ef4a8f34f1f84553bd70090f022 (127.10.98.1:35553)
I20260926 12:33:35.794402 13690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:34405
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
I20260926 12:33:35.801052 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:36501
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:41961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:35.910597 13887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:35.910871 13887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:35.910914 13887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:35.914897 13887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:35.914959 13887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:35.915019 13887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:33:35.920262 13887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:41961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:36501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:35.921293 13887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:35.922258 13887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:35.931583 13892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:35.931612 13893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:35.931578 13895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:35.931953 13887 server_base.cc:1034] running on GCE node
I20260926 12:33:35.932380 13887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:35.933015 13887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:35.934180 13887 hybrid_clock.cc:648] HybridClock initialized: now 1790426015934155 us; error 35 us; skew 500 ppm
Sep 26 12:33:35 dist-test-slave-t7tk krb5kdc[13646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426015, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:35.937029 13887 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:33:35.937911 13887 webserver.cc:460] Webserver started at http://127.10.98.2:33831/ using document root <none> and password file <none>
I20260926 12:33:35.938426 13887 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:35.938483 13887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:35.938647 13887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:35.940682 13887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "45af466ae9474f85a5e34934a1136787"
format_stamp: "Formatted at 2026-09-26 12:33:35 on dist-test-slave-t7tk"
server_key: "2f586641b8284c1e731fd8b44a32143a"
server_key_iv: "edce1ff869f7ff755108c7362829fc49"
server_key_version: "encryptionkey@0"
I20260926 12:33:35.941255 13887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "45af466ae9474f85a5e34934a1136787"
format_stamp: "Formatted at 2026-09-26 12:33:35 on dist-test-slave-t7tk"
server_key: "2f586641b8284c1e731fd8b44a32143a"
server_key_iv: "edce1ff869f7ff755108c7362829fc49"
server_key_version: "encryptionkey@0"
I20260926 12:33:35.944988 13887 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:33:35.947474 13902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:35.948721 13887 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:33:35.948885 13887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "45af466ae9474f85a5e34934a1136787"
format_stamp: "Formatted at 2026-09-26 12:33:35 on dist-test-slave-t7tk"
server_key: "2f586641b8284c1e731fd8b44a32143a"
server_key_iv: "edce1ff869f7ff755108c7362829fc49"
server_key_version: "encryptionkey@0"
I20260926 12:33:35.948992 13887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:35.964498 13887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:35.967569 13887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:35.967747 13887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:35.968327 13887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:35.969154 13887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:35.969209 13887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:35.969270 13887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:35.969295 13887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:35.977706 13887 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:36693
I20260926 12:33:35.977736 14015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:36693 every 8 connection(s)
I20260926 12:33:35.978578 13887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:33:35.987504 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 13887
I20260926 12:33:35.987609 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:33:35.987861 10632 external_mini_cluster.cc:1428] Setting key 05724c6b920266345935f29e60183e10
Sep 26 12:33:35 dist-test-slave-t7tk krb5kdc[13646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426015, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:35.992223 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:35.980423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:35869 (local address 127.10.98.62:36501)
0926 12:33:35.980578 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:35.980581 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:35.981439 (+ 858us) server_negotiation.cc:407] Connection header received
0926 12:33:35.982815 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:35.982818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:35.982864 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:35.982937 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:35.984094 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:35.984610 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:35.985806 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:35.986034 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:35.986113 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:35.988602 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:35.988606 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:35.988609 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:35.988633 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:35.990103 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:35.990723 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:35.990727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:35.990729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:35.990839 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:35.991158 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:35.991161 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:35.991163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:35.991393 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:35.991490 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:35.991952 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 12:33:35.992067 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 12:33:35.993165 14018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:35.980784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:36501 (local address 127.10.98.2:35869)
0926 12:33:35.981283 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:35.981326 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:35.982614 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:35.983130 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:35.983137 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:35.983492 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:35.983971 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:35.983984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:35.984715 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:35.984718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:35.985690 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:35.985700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:35.986188 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:35.986192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:35.986344 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:35.986938 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:35.986954 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:35.988448 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:35.990220 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:35.990227 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:35.990243 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:35.990579 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:35.990943 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:35.990945 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:35.990947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:35.991080 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:35.991483 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:35.991488 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:35.991725 (+ 237us) client_negotiation.cc:769] Sending connection context
0926 12:33:35.991880 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 12:33:35.992196 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":115,"threads_started":1}
I20260926 12:33:35.994357 14016 heartbeater.cc:347] Connected to a master server at 127.10.98.62:36501
I20260926 12:33:35.994629 14016 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:35.995234 14016 heartbeater.cc:511] Master 127.10.98.62:36501 requested a full tablet report, sending...
I20260926 12:33:35.996389 13690 ts_manager.cc:194] Registered new tserver with Master: 45af466ae9474f85a5e34934a1136787 (127.10.98.2:36693)
I20260926 12:33:35.996971 13690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:35869
WARNING: no policy specified for kudu/127.10.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
I20260926 12:33:36.016582 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:36501
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:41961
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:36.125051 14021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:36.125389 14021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:36.125440 14021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:36.129492 14021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:36.129551 14021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:36.129619 14021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:33:36.134887 14021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:41961
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:36501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:36.135934 14021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:36.136951 14021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:36.146584 14026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:36.146540 14029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:36.146606 14021 server_base.cc:1034] running on GCE node
W20260926 12:33:36.146548 14027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:36.147241 14021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:36.147876 14021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:36.149039 14021 hybrid_clock.cc:648] HybridClock initialized: now 1790426016149012 us; error 40 us; skew 500 ppm
Sep 26 12:33:36 dist-test-slave-t7tk krb5kdc[13646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426016, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:36.151716 14021 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:33:36.152704 14021 webserver.cc:460] Webserver started at http://127.10.98.3:36383/ using document root <none> and password file <none>
I20260926 12:33:36.153321 14021 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:36.153394 14021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:36.153591 14021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:36.155637 14021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "46f74e5b87544d15b14cd696297a40fd"
format_stamp: "Formatted at 2026-09-26 12:33:36 on dist-test-slave-t7tk"
server_key: "1ea30312fc4693bb7c487c2a6e6bb34c"
server_key_iv: "6a743740d392bd555fd86c0ef01c6117"
server_key_version: "encryptionkey@0"
I20260926 12:33:36.156176 14021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "46f74e5b87544d15b14cd696297a40fd"
format_stamp: "Formatted at 2026-09-26 12:33:36 on dist-test-slave-t7tk"
server_key: "1ea30312fc4693bb7c487c2a6e6bb34c"
server_key_iv: "6a743740d392bd555fd86c0ef01c6117"
server_key_version: "encryptionkey@0"
I20260926 12:33:36.159977 14021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260926 12:33:36.162640 14036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:36.163985 14021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:33:36.164104 14021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "46f74e5b87544d15b14cd696297a40fd"
format_stamp: "Formatted at 2026-09-26 12:33:36 on dist-test-slave-t7tk"
server_key: "1ea30312fc4693bb7c487c2a6e6bb34c"
server_key_iv: "6a743740d392bd555fd86c0ef01c6117"
server_key_version: "encryptionkey@0"
I20260926 12:33:36.164209 14021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:36.180047 14021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:36.183251 14021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:36.183441 14021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:36.184058 14021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:36.184890 14021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:36.184945 14021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:36.185000 14021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:36.185029 14021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:36.193799 14021 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:40891
I20260926 12:33:36.193888 14149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:40891 every 8 connection(s)
I20260926 12:33:36.194737 14021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:33:36.203007 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 14021
I20260926 12:33:36.203145 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:33:36.203466 10632 external_mini_cluster.cc:1428] Setting key 34892938d66cb9915662560044419966
Sep 26 12:33:36 dist-test-slave-t7tk krb5kdc[13646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426016, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:36.208683 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:36.196620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:33981 (local address 127.10.98.62:36501)
0926 12:33:36.196754 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:36.196758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:36.197666 (+ 908us) server_negotiation.cc:407] Connection header received
0926 12:33:36.199058 (+ 1392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:36.199061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:36.199113 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:36.199192 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:36.200409 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:36.201035 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:36.202231 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:36.202459 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:36.202545 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:36.205419 (+ 2874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:36.205423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:36.205426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:36.205456 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:36.206732 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:36.207230 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:36.207234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:36.207236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:36.207313 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:36.207628 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:36.207631 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:36.207633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:36.207812 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:36.207904 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:36.208396 (+ 492us) server_negotiation.cc:299] Negotiation successful
0926 12:33:36.208523 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:33:36.209556 14152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:36.197020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:36501 (local address 127.10.98.3:33981)
0926 12:33:36.197506 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:36.197551 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:36.198806 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:36.199338 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:36.199345 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:36.199713 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:36.200240 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:36.200255 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:36.201191 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:36.201193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:36.202108 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:36.202117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:36.202591 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:36.202594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:36.202752 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:36.203472 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:36.203494 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:36.205258 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:36.206845 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:36.206851 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:36.206868 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:36.207137 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:36.207400 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:36.207403 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:36.207405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:36.207545 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:36.207909 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:36.207917 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:36.208171 (+ 254us) client_negotiation.cc:769] Sending connection context
0926 12:33:36.208341 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 12:33:36.208690 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":124,"threads_started":1}
I20260926 12:33:36.210602 14150 heartbeater.cc:347] Connected to a master server at 127.10.98.62:36501
I20260926 12:33:36.210813 14150 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:36.211282 14150 heartbeater.cc:511] Master 127.10.98.62:36501 requested a full tablet report, sending...
I20260926 12:33:36.212253 13690 ts_manager.cc:194] Registered new tserver with Master: 46f74e5b87544d15b14cd696297a40fd (127.10.98.3:40891)
I20260926 12:33:36.212790 13690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:33981
I20260926 12:33:36.217835 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 12:33:36 dist-test-slave-t7tk krb5kdc[13646](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426016, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 12:33:36 dist-test-slave-t7tk krb5kdc[13646](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426016, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:36.243513 13730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:36.234220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44028 (local address 127.10.98.62:36501)
0926 12:33:36.234355 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:36.234363 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:36.234532 (+ 169us) server_negotiation.cc:407] Connection header received
0926 12:33:36.234694 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:36.234697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:36.234744 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:36.234905 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:36.235599 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:36.236128 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:36.237265 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:36.237578 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:36.237690 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:36.240534 (+ 2844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:36.240539 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:36.240541 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:36.240569 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:36.241883 (+ 1314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:36.242291 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:36.242294 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:36.242296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:36.242384 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:36.242630 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:36.242633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:36.242635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:36.242874 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:36.243006 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:36.243240 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 12:33:36.243356 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 12:33:36.243973 13690 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44028
I20260926 12:33:36.245127 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 13753
I20260926 12:33:36.250778 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 13887
I20260926 12:33:36.255851 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 14021
I20260926 12:33:36.260772 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 13659
2026-09-26T12:33:36Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2045 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:33:36 dist-test-slave-t7tk krb5kdc[14162](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:33:36 dist-test-slave-t7tk krb5kdc[14162](info): set up 2 sockets
Sep 26 12:33:36 dist-test-slave-t7tk krb5kdc[14162](info): commencing operation
krb5kdc: starting...
W20260926 12:33:37.279633 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 12:33:37.517189 10632 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.237s user 0.002s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260926 12:33:37.810024 10632 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.293s user 0.001s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260926 12:33:38.032130 10632 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.222s user 0.001s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/test-user.keytab.
W20260926 12:33:38.977160 10632 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.945s user 0.001s sys 0.001s
Sep 26 12:33:38 dist-test-slave-t7tk krb5kdc[14162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:33:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:33:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
W20260926 12:33:39.958293 10632 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.10.98.62: real 0.959s user 0.000s sys 0.003s
I20260926 12:33:39.958791 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:37183
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:37183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:40.075392 14175 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:40.075659 14175 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:40.075701 14175 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:40.079720 14175 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:40.079785 14175 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:40.079803 14175 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:40.079818 14175 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:40.079833 14175 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:33:40.085232 14175 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:37183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:37183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14175
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:40.086350 14175 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:40.087404 14175 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:40.096050 14183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:40.096079 14180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:40.096114 14181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:40.096433 14175 server_base.cc:1034] running on GCE node
I20260926 12:33:40.097039 14175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:40.098492 14175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:40.099699 14175 hybrid_clock.cc:648] HybridClock initialized: now 1790426020099702 us; error 76 us; skew 500 ppm
Sep 26 12:33:40 dist-test-slave-t7tk krb5kdc[14162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426020, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:40.109421 14175 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:33:40.110532 14175 webserver.cc:460] Webserver started at http://127.10.98.62:34655/ using document root <none> and password file <none>
I20260926 12:33:40.111159 14175 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:40.111231 14175 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:40.111442 14175 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:40.113719 14175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "453016d978fd408b966815832772d3fd"
format_stamp: "Formatted at 2026-09-26 12:33:40 on dist-test-slave-t7tk"
server_key: "2bc32e7b09349c13e32a01127e9375d9"
server_key_iv: "73efa997d20afc29690a1397b8909a12"
server_key_version: "encryptionkey@0"
I20260926 12:33:40.114291 14175 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "453016d978fd408b966815832772d3fd"
format_stamp: "Formatted at 2026-09-26 12:33:40 on dist-test-slave-t7tk"
server_key: "2bc32e7b09349c13e32a01127e9375d9"
server_key_iv: "73efa997d20afc29690a1397b8909a12"
server_key_version: "encryptionkey@0"
I20260926 12:33:40.118567 14175 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:33:40.121393 14190 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:40.122845 14175 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:33:40.123026 14175 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "453016d978fd408b966815832772d3fd"
format_stamp: "Formatted at 2026-09-26 12:33:40 on dist-test-slave-t7tk"
server_key: "2bc32e7b09349c13e32a01127e9375d9"
server_key_iv: "73efa997d20afc29690a1397b8909a12"
server_key_version: "encryptionkey@0"
I20260926 12:33:40.123136 14175 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:40.147814 14175 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:40.151089 14175 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:40.151274 14175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:40.158243 14175 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:37183
I20260926 12:33:40.158288 14242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:37183 every 8 connection(s)
I20260926 12:33:40.159240 14175 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:33:40.162150 14243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:40.165405 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 14175
I20260926 12:33:40.165537 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:33:40.165774 10632 external_mini_cluster.cc:1428] Setting key 01e90451231eb639c9002b3854b95ff3
I20260926 12:33:40.168257 14243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd: Bootstrap starting.
I20260926 12:33:40.171051 14243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:40.171906 14243 log.cc:824] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:40.174153 14243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd: No bootstrap required, opened a new log
Sep 26 12:33:40 dist-test-slave-t7tk krb5kdc[14162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:40.177754 14243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453016d978fd408b966815832772d3fd" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 37183 } }
I20260926 12:33:40.178040 14243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:40.178103 14243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 453016d978fd408b966815832772d3fd, State: Initialized, Role: FOLLOWER
I20260926 12:33:40.178633 14243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453016d978fd408b966815832772d3fd" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 37183 } }
I20260926 12:33:40.178774 14243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:33:40.178833 14243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:33:40.178929 14243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:40.181542 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:40.167244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44454 (local address 127.10.98.62:37183)
0926 12:33:40.167709 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:40.167728 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:40.167772 (+ 44us) server_negotiation.cc:407] Connection header received
0926 12:33:40.168572 (+ 800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:40.168595 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:40.169022 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:40.169436 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:40.170124 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:40.171591 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:40.172600 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:40.173396 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:40.173552 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:40.175910 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:40.175916 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:40.175933 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:40.175964 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:40.178479 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:40.178986 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:40.178993 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:40.179000 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:40.179142 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:40.179425 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:40.179428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:40.179430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:40.179937 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:40.180121 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:40.180414 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 12:33:40.180720 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":118,"threads_started":1}
I20260926 12:33:40.227901 14243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453016d978fd408b966815832772d3fd" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 37183 } }
I20260926 12:33:40.228452 14243 leader_election.cc:302] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 453016d978fd408b966815832772d3fd; no voters:
I20260926 12:33:40.228863 14243 leader_election.cc:288] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:33:40.229007 14248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:40.229271 14248 raft_consensus.cc:686] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd [term 1 LEADER]: Becoming Leader. State: Replica: 453016d978fd408b966815832772d3fd, State: Running, Role: LEADER
I20260926 12:33:40.229676 14248 consensus_queue.cc:237] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453016d978fd408b966815832772d3fd" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 37183 } }
I20260926 12:33:40.230365 14243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:40.231382 14249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "453016d978fd408b966815832772d3fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453016d978fd408b966815832772d3fd" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 37183 } } }
I20260926 12:33:40.231520 14249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:40.231868 14257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:33:40.231966 14250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 453016d978fd408b966815832772d3fd. Latest consensus state: current_term: 1 leader_uuid: "453016d978fd408b966815832772d3fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "453016d978fd408b966815832772d3fd" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 37183 } } }
I20260926 12:33:40.232136 14250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:40.235196 14257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:33:40.241544 14257 catalog_manager.cc:1347] Generated new cluster ID: 21ddf1b92e434d19babd29eaeff0b81c
I20260926 12:33:40.241616 14257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:33:40.262044 14257 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:33:40.263479 14257 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:33:40.280684 14257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 453016d978fd408b966815832772d3fd: Generated new TSK 0
I20260926 12:33:40.281541 14257 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
W20260926 12:33:41.048974 10632 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.10.98.1: real 0.759s user 0.002s sys 0.002s
I20260926 12:33:41.049551 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:37183
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:41.165745 14269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:41.166132 14269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:41.166198 14269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:41.171969 14269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:41.172052 14269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:41.172116 14269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:33:41.177675 14269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:37183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:41.178925 14269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:41.180315 14269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:41.190181 14274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:41.190567 14269 server_base.cc:1034] running on GCE node
W20260926 12:33:41.191895 14275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:41.192054 14277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:41.192512 14269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:41.193243 14269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:41.194473 14269 hybrid_clock.cc:648] HybridClock initialized: now 1790426021194447 us; error 37 us; skew 500 ppm
Sep 26 12:33:41 dist-test-slave-t7tk krb5kdc[14162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426021, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:41.197510 14269 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:33:41.198529 14269 webserver.cc:460] Webserver started at http://127.10.98.1:40209/ using document root <none> and password file <none>
I20260926 12:33:41.199103 14269 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:41.199168 14269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:41.199395 14269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:41.201599 14269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "a211490d651f4cfe97e7dd6237dee985"
format_stamp: "Formatted at 2026-09-26 12:33:41 on dist-test-slave-t7tk"
server_key: "272cc681d29ab6c795c1a21ccacee173"
server_key_iv: "8bec6291e1b369ec4a214f94efb31eaa"
server_key_version: "encryptionkey@0"
I20260926 12:33:41.202170 14269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "a211490d651f4cfe97e7dd6237dee985"
format_stamp: "Formatted at 2026-09-26 12:33:41 on dist-test-slave-t7tk"
server_key: "272cc681d29ab6c795c1a21ccacee173"
server_key_iv: "8bec6291e1b369ec4a214f94efb31eaa"
server_key_version: "encryptionkey@0"
I20260926 12:33:41.206218 14269 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 12:33:41.209275 14284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:41.210707 14269 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:33:41.210848 14269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "a211490d651f4cfe97e7dd6237dee985"
format_stamp: "Formatted at 2026-09-26 12:33:41 on dist-test-slave-t7tk"
server_key: "272cc681d29ab6c795c1a21ccacee173"
server_key_iv: "8bec6291e1b369ec4a214f94efb31eaa"
server_key_version: "encryptionkey@0"
I20260926 12:33:41.210968 14269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:41.242089 14269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:41.245563 14269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:41.245817 14269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:41.246485 14269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:41.247350 14269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:41.247399 14269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:41.247453 14269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:41.247478 14269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:41.256985 14269 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:36215
I20260926 12:33:41.257079 14397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:36215 every 8 connection(s)
I20260926 12:33:41.258220 14269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:33:41.266655 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 14269
I20260926 12:33:41.266785 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:33:41.267053 10632 external_mini_cluster.cc:1428] Setting key 0d06ecabf8b09cedbfeb8836e0e4cb59
WARNING: no policy specified for kudu/127.10.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.2@KRBTEST.COM" created.
Sep 26 12:33:41 dist-test-slave-t7tk krb5kdc[14162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426021, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:41.283972 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:41.267926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:48319 (local address 127.10.98.62:37183)
0926 12:33:41.269697 (+ 1771us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:41.269701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:41.269717 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:33:41.271010 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:41.271014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:41.271075 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:41.271175 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:41.272646 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:41.273529 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:41.275153 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:41.275528 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:41.275644 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:41.280235 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:41.280239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:41.280241 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:41.280270 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:41.281825 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:41.282377 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:41.282380 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:41.282381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:41.282464 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:41.282810 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:41.282814 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:41.282815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:41.283010 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:41.283097 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:41.283646 (+ 549us) server_negotiation.cc:299] Negotiation successful
0926 12:33:41.283791 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1693,"thread_start_us":86,"threads_started":1}
I20260926 12:33:41.284914 14401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:41.268511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:37183 (local address 127.10.98.1:48319)
0926 12:33:41.269122 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:41.269196 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:41.270755 (+ 1559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:41.271363 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:41.271373 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:41.271844 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:41.272487 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:41.272504 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:41.273650 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:41.273653 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:41.275018 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:41.275029 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:41.275683 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:41.275686 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:41.275882 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:41.276541 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:41.276563 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:41.280017 (+ 3454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:41.281957 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:41.281964 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:41.281984 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:41.282272 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:41.282550 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:41.282553 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:41.282556 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:41.282720 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:41.283117 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:41.283124 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:41.283408 (+ 284us) client_negotiation.cc:769] Sending connection context
0926 12:33:41.283588 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 12:33:41.283966 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":148,"threads_started":1}
I20260926 12:33:41.286396 14398 heartbeater.cc:347] Connected to a master server at 127.10.98.62:37183
I20260926 12:33:41.286641 14398 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:41.287177 14398 heartbeater.cc:511] Master 127.10.98.62:37183 requested a full tablet report, sending...
I20260926 12:33:41.288921 14207 ts_manager.cc:194] Registered new tserver with Master: a211490d651f4cfe97e7dd6237dee985 (127.10.98.1:36215)
I20260926 12:33:41.290127 14207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:48319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
I20260926 12:33:41.297142 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:37183
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:41.413225 14404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:41.413506 14404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:41.413549 14404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:41.417747 14404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:41.417821 14404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:41.417886 14404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:33:41.423350 14404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:37183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:41.424449 14404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:41.425555 14404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:41.435652 14412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:41.435693 14409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:41.435778 14404 server_base.cc:1034] running on GCE node
W20260926 12:33:41.435681 14410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:41.436501 14404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:41.437199 14404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:41.438370 14404 hybrid_clock.cc:648] HybridClock initialized: now 1790426021438343 us; error 41 us; skew 500 ppm
Sep 26 12:33:41 dist-test-slave-t7tk krb5kdc[14162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426021, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:41.441190 14404 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:33:41.442117 14404 webserver.cc:460] Webserver started at http://127.10.98.2:40659/ using document root <none> and password file <none>
I20260926 12:33:41.442687 14404 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:41.442749 14404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:41.442961 14404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:41.445039 14404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "f0274f6f452b48109cf28210f861711e"
format_stamp: "Formatted at 2026-09-26 12:33:41 on dist-test-slave-t7tk"
server_key: "e1123a733e73f4d94ae3aa2b939aeb48"
server_key_iv: "27de2a14aadd6934ed9ee146095897fa"
server_key_version: "encryptionkey@0"
I20260926 12:33:41.445672 14404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "f0274f6f452b48109cf28210f861711e"
format_stamp: "Formatted at 2026-09-26 12:33:41 on dist-test-slave-t7tk"
server_key: "e1123a733e73f4d94ae3aa2b939aeb48"
server_key_iv: "27de2a14aadd6934ed9ee146095897fa"
server_key_version: "encryptionkey@0"
I20260926 12:33:41.449666 14404 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:33:41.452342 14419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:41.453895 14404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:33:41.454028 14404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "f0274f6f452b48109cf28210f861711e"
format_stamp: "Formatted at 2026-09-26 12:33:41 on dist-test-slave-t7tk"
server_key: "e1123a733e73f4d94ae3aa2b939aeb48"
server_key_iv: "27de2a14aadd6934ed9ee146095897fa"
server_key_version: "encryptionkey@0"
I20260926 12:33:41.454154 14404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:41.482247 14404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:41.485729 14404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:41.485924 14404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:41.486549 14404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:41.487414 14404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:41.487464 14404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:41.487519 14404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:41.487548 14404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:41.497613 14404 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:34649
I20260926 12:33:41.497660 14532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:34649 every 8 connection(s)
I20260926 12:33:41.498668 14404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:33:41.504338 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 14404
I20260926 12:33:41.504465 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:33:41.504725 10632 external_mini_cluster.cc:1428] Setting key cb3810591459def360c98001b9b0c162
Sep 26 12:33:41 dist-test-slave-t7tk krb5kdc[14162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426021, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:41.514727 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:41.500743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:43633 (local address 127.10.98.62:37183)
0926 12:33:41.500925 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:41.500929 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:41.501865 (+ 936us) server_negotiation.cc:407] Connection header received
0926 12:33:41.503387 (+ 1522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:41.503391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:41.503442 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:41.503537 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:41.504936 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:41.505662 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:41.507261 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:41.507615 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:41.507743 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:41.510334 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:41.510339 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:41.510343 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:41.510379 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:41.512079 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:41.512805 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:41.512810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:41.512812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:41.512920 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:41.513358 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:41.513362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:41.513364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:41.513651 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:41.513775 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:41.514398 (+ 623us) server_negotiation.cc:299] Negotiation successful
0926 12:33:41.514549 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
WARNING: no policy specified for kudu/127.10.98.3@KRBTEST.COM; defaulting to no policy
I20260926 12:33:41.515681 14535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:41.501181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:37183 (local address 127.10.98.2:43633)
0926 12:33:41.501689 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:41.501747 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:41.503101 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:41.503717 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:41.503724 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:41.504112 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:41.504741 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:41.504760 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:41.505878 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:41.505882 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:41.507094 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:41.507106 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:41.507795 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:41.507799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:41.507993 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:41.508609 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:41.508627 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:41.510129 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:41.512283 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:41.512289 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:41.512306 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:41.512618 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:41.513042 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:41.513045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:41.513046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:41.513258 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:41.513781 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:41.513787 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:41.514073 (+ 286us) client_negotiation.cc:769] Sending connection context
0926 12:33:41.514252 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 12:33:41.514631 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":110,"threads_started":1}
I20260926 12:33:41.517045 14533 heartbeater.cc:347] Connected to a master server at 127.10.98.62:37183
I20260926 12:33:41.517349 14533 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:41.517948 14533 heartbeater.cc:511] Master 127.10.98.62:37183 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.3@KRBTEST.COM" created.
I20260926 12:33:41.519187 14207 ts_manager.cc:194] Registered new tserver with Master: f0274f6f452b48109cf28210f861711e (127.10.98.2:34649)
I20260926 12:33:41.519852 14207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:43633
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
I20260926 12:33:41.538246 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:37183
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:41343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:41.663106 14538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:41.663396 14538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:41.663441 14538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:41.667661 14538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:41.667739 14538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:41.667801 14538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:33:41.673725 14538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:41343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:37183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:41.674880 14538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:41.675948 14538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:41.686764 14544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:41.686822 14543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:41.686887 14546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:41.686864 14538 server_base.cc:1034] running on GCE node
I20260926 12:33:41.687583 14538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:41.688382 14538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:41.689560 14538 hybrid_clock.cc:648] HybridClock initialized: now 1790426021689525 us; error 51 us; skew 500 ppm
Sep 26 12:33:41 dist-test-slave-t7tk krb5kdc[14162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426021, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:41.692510 14538 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:33:41.693480 14538 webserver.cc:460] Webserver started at http://127.10.98.3:41697/ using document root <none> and password file <none>
I20260926 12:33:41.694036 14538 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:41.694104 14538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:41.694322 14538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:41.696942 14538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "53a5113af83d43d58aee7c685438fa0c"
format_stamp: "Formatted at 2026-09-26 12:33:41 on dist-test-slave-t7tk"
server_key: "dc4ff61d741955ad90a7c962e46e401f"
server_key_iv: "42e27c71ecc9c076c861ba8a06ee5b4f"
server_key_version: "encryptionkey@0"
I20260926 12:33:41.697616 14538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "53a5113af83d43d58aee7c685438fa0c"
format_stamp: "Formatted at 2026-09-26 12:33:41 on dist-test-slave-t7tk"
server_key: "dc4ff61d741955ad90a7c962e46e401f"
server_key_iv: "42e27c71ecc9c076c861ba8a06ee5b4f"
server_key_version: "encryptionkey@0"
I20260926 12:33:41.701757 14538 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:33:41.704568 14553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:41.706161 14538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:33:41.706296 14538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "53a5113af83d43d58aee7c685438fa0c"
format_stamp: "Formatted at 2026-09-26 12:33:41 on dist-test-slave-t7tk"
server_key: "dc4ff61d741955ad90a7c962e46e401f"
server_key_iv: "42e27c71ecc9c076c861ba8a06ee5b4f"
server_key_version: "encryptionkey@0"
I20260926 12:33:41.706435 14538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:41.728173 14538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:41.731722 14538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:41.731938 14538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:41.732553 14538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:41.733433 14538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:41.733479 14538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:41.733531 14538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:41.733559 14538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:41.742048 14538 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:38117
I20260926 12:33:41.742066 14666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:38117 every 8 connection(s)
I20260926 12:33:41.742991 14538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:33:41.745448 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 14538
I20260926 12:33:41.745575 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:33:41.745929 10632 external_mini_cluster.cc:1428] Setting key f665dc375e337f87ba8de348ce446a35
Sep 26 12:33:41 dist-test-slave-t7tk krb5kdc[14162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426021, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:41.757769 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:41.744909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:33005 (local address 127.10.98.62:37183)
0926 12:33:41.745064 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:41.745068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:41.746306 (+ 1238us) server_negotiation.cc:407] Connection header received
0926 12:33:41.748017 (+ 1711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:41.748020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:41.748072 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:41.748144 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:41.749348 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:41.750048 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:41.751197 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:41.751444 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:41.751525 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:41.754087 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:41.754091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:41.754093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:41.754119 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:41.755489 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:41.755998 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:41.756001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:41.756002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:41.756080 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:41.756495 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:41.756498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:41.756499 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:41.756687 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:41.756764 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:41.757482 (+ 718us) server_negotiation.cc:299] Negotiation successful
0926 12:33:41.757595 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 12:33:41.758822 14669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:41.745494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:37183 (local address 127.10.98.3:33005)
0926 12:33:41.746129 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:41.746179 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:41.747774 (+ 1595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:41.748302 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:41.748309 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:41.748672 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:41.749224 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:41.749239 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:41.750176 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:41.750178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:41.751076 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:41.751085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:41.751567 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:41.751569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:41.751718 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:41.752333 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:41.752350 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:41.753923 (+ 1573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:41.755610 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:41.755619 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:41.755633 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:41.755897 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:41.756181 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:41.756184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:41.756185 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:41.756357 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:41.756848 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:41.756854 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:41.757167 (+ 313us) client_negotiation.cc:769] Sending connection context
0926 12:33:41.757386 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 12:33:41.757807 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":145,"threads_started":1}
I20260926 12:33:41.759970 14667 heartbeater.cc:347] Connected to a master server at 127.10.98.62:37183
I20260926 12:33:41.760223 14667 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:41.760731 14667 heartbeater.cc:511] Master 127.10.98.62:37183 requested a full tablet report, sending...
I20260926 12:33:41.761775 14207 ts_manager.cc:194] Registered new tserver with Master: 53a5113af83d43d58aee7c685438fa0c (127.10.98.3:38117)
I20260926 12:33:41.762367 14207 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:33005
I20260926 12:33:41.772127 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 12:33:41 dist-test-slave-t7tk krb5kdc[14162](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 12:33:41 dist-test-slave-t7tk krb5kdc[14162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.1@KRBTEST.COM
I20260926 12:33:41.798666 14674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:41.788285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58608 (local address 127.10.98.1:36215)
0926 12:33:41.788595 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:41.788603 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:41.788629 (+ 26us) server_negotiation.cc:407] Connection header received
0926 12:33:41.788690 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:41.788697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:41.788937 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:41.789072 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:41.789823 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:41.790639 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:41.791488 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:41.791874 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:41.791967 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:41.794618 (+ 2651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:41.794629 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:41.794648 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:41.794678 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:41.796466 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:41.796985 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:41.796991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:41.796992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:41.797088 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:41.797451 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:41.797454 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:41.797455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:41.797825 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:41.798004 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:41.798287 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 12:33:41.798499 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":88,"threads_started":1}
Sep 26 12:33:41 dist-test-slave-t7tk krb5kdc[14162](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426021, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:41.810395 14400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:41.801905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44460 (local address 127.10.98.62:37183)
0926 12:33:41.802051 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:41.802056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:41.802073 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:33:41.802210 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:41.802213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:41.802260 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:41.802351 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:41.802943 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:41.803472 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:41.804264 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:41.804500 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:41.804588 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:41.807075 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:41.807079 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:41.807081 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:41.807109 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:41.808571 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:41.809052 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:41.809056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:41.809058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:41.809183 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:41.809438 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:41.809442 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:41.809444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:41.809689 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:41.809807 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:41.810033 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 12:33:41.810156 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260926 12:33:41.810986 14207 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44460
I20260926 12:33:41.811682 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 14269
I20260926 12:33:41.817351 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 14404
I20260926 12:33:41.822221 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 14538
I20260926 12:33:41.827087 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 14175
2026-09-26T12:33:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (5566 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:33:41 dist-test-slave-t7tk krb5kdc[14679](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:33:41 dist-test-slave-t7tk krb5kdc[14679](info): set up 2 sockets
Sep 26 12:33:41 dist-test-slave-t7tk krb5kdc[14679](info): commencing operation
krb5kdc: starting...
W20260926 12:33:42.790621 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:33:42 dist-test-slave-t7tk krb5kdc[14679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:33:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:33:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:33:42.917613 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:40903
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:40903
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:43.027218 14692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:43.027525 14692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:43.027586 14692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:43.031697 14692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:43.031772 14692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:43.031806 14692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:43.031827 14692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:43.031847 14692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:33:43.037247 14692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:40903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:40903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:43.038460 14692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:43.039494 14692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:43.047936 14697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:43.047963 14698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:43.047945 14700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:43.048238 14692 server_base.cc:1034] running on GCE node
I20260926 12:33:43.048905 14692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:43.050140 14692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:43.051318 14692 hybrid_clock.cc:648] HybridClock initialized: now 1790426023051299 us; error 36 us; skew 500 ppm
Sep 26 12:33:43 dist-test-slave-t7tk krb5kdc[14679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426023, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:43.054179 14692 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:33:43.054947 14692 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:43.055014 14692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:43.055181 14692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:43.057257 14692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "a6e361b1ac2946f0adbf29b8368962d2"
format_stamp: "Formatted at 2026-09-26 12:33:43 on dist-test-slave-t7tk"
server_key: "ca1dc00be2f19a9d0582af57b4a32a93"
server_key_iv: "9be148cde7db73f48c49070b817798a2"
server_key_version: "encryptionkey@0"
I20260926 12:33:43.057768 14692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "a6e361b1ac2946f0adbf29b8368962d2"
format_stamp: "Formatted at 2026-09-26 12:33:43 on dist-test-slave-t7tk"
server_key: "ca1dc00be2f19a9d0582af57b4a32a93"
server_key_iv: "9be148cde7db73f48c49070b817798a2"
server_key_version: "encryptionkey@0"
I20260926 12:33:43.061441 14692 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:33:43.064009 14706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:43.065443 14692 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:33:43.065552 14692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "a6e361b1ac2946f0adbf29b8368962d2"
format_stamp: "Formatted at 2026-09-26 12:33:43 on dist-test-slave-t7tk"
server_key: "ca1dc00be2f19a9d0582af57b4a32a93"
server_key_iv: "9be148cde7db73f48c49070b817798a2"
server_key_version: "encryptionkey@0"
I20260926 12:33:43.065645 14692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:43.106675 14692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:43.109965 14692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:43.110134 14692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:43.117087 14692 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:40903
I20260926 12:33:43.117439 14758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:40903 every 8 connection(s)
I20260926 12:33:43.117795 14692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:33:43.120882 14759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:43.124598 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 14692
I20260926 12:33:43.124732 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:33:43.125013 10632 external_mini_cluster.cc:1428] Setting key e037ea21c8dbb0b72fa8857d9e8900b9
I20260926 12:33:43.127317 14759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2: Bootstrap starting.
I20260926 12:33:43.129968 14759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:43.130756 14759 log.cc:824] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:43.133165 14759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2: No bootstrap required, opened a new log
Sep 26 12:33:43 dist-test-slave-t7tk krb5kdc[14679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:43.136924 14759 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e361b1ac2946f0adbf29b8368962d2" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40903 } }
I20260926 12:33:43.137202 14759 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:43.137275 14759 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e361b1ac2946f0adbf29b8368962d2, State: Initialized, Role: FOLLOWER
I20260926 12:33:43.137801 14759 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e361b1ac2946f0adbf29b8368962d2" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40903 } }
I20260926 12:33:43.137943 14759 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:33:43.138012 14759 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:33:43.138110 14759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:43.139199 14759 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e361b1ac2946f0adbf29b8368962d2" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40903 } }
I20260926 12:33:43.139539 14759 leader_election.cc:302] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6e361b1ac2946f0adbf29b8368962d2; no voters:
I20260926 12:33:43.139700 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:43.126641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34658 (local address 127.10.98.62:40903)
0926 12:33:43.127064 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:43.127080 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:43.127116 (+ 36us) server_negotiation.cc:407] Connection header received
0926 12:33:43.127826 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:43.127848 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:43.128233 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:43.128533 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:43.129483 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.130681 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.131701 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.132184 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.132317 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:43.135007 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:43.135013 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:43.135028 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:43.135057 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:43.136680 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:43.137131 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:43.137138 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:43.137145 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:43.137280 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:43.137598 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:43.137602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:43.137604 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:43.138062 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:43.138259 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:43.138520 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 12:33:43.138812 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":117,"threads_started":1}
I20260926 12:33:43.139870 14759 leader_election.cc:288] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:33:43.139966 14764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:43.140172 14764 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2 [term 1 LEADER]: Becoming Leader. State: Replica: a6e361b1ac2946f0adbf29b8368962d2, State: Running, Role: LEADER
I20260926 12:33:43.140565 14764 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e361b1ac2946f0adbf29b8368962d2" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40903 } }
I20260926 12:33:43.141242 14759 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:43.142637 14766 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6e361b1ac2946f0adbf29b8368962d2. Latest consensus state: current_term: 1 leader_uuid: "a6e361b1ac2946f0adbf29b8368962d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e361b1ac2946f0adbf29b8368962d2" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40903 } } }
I20260926 12:33:43.142773 14766 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:43.143532 14775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:33:43.143621 14765 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6e361b1ac2946f0adbf29b8368962d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6e361b1ac2946f0adbf29b8368962d2" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40903 } } }
I20260926 12:33:43.143898 14765 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:43.146579 14775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:33:43.152544 14775 catalog_manager.cc:1347] Generated new cluster ID: 450dc543ac1445c281d76be8a261ae08
I20260926 12:33:43.152631 14775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:33:43.165186 14775 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:33:43.166543 14775 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:33:43.174015 14775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2: Generated new TSK 0
I20260926 12:33:43.174808 14775 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
I20260926 12:33:43.209450 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:40903
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:43.323642 14785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:43.323957 14785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:43.324003 14785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:43.328122 14785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:43.328198 14785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:43.328260 14785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:33:43.333810 14785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:40903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:43.334957 14785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:43.335997 14785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:43.345551 14791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:43.345547 14790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:43.345829 14793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:43.345976 14785 server_base.cc:1034] running on GCE node
I20260926 12:33:43.346398 14785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:43.347005 14785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:43.348165 14785 hybrid_clock.cc:648] HybridClock initialized: now 1790426023348145 us; error 31 us; skew 500 ppm
Sep 26 12:33:43 dist-test-slave-t7tk krb5kdc[14679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426023, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:43.351084 14785 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:33:43.351980 14785 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:43.352072 14785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:43.352280 14785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:43.354396 14785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "6e0957a0067c423b997e4e207381e710"
format_stamp: "Formatted at 2026-09-26 12:33:43 on dist-test-slave-t7tk"
server_key: "85daadb1df5067380c6771118d0957b6"
server_key_iv: "4ce08c71235a0b52314899d76e7f7cf5"
server_key_version: "encryptionkey@0"
I20260926 12:33:43.354959 14785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "6e0957a0067c423b997e4e207381e710"
format_stamp: "Formatted at 2026-09-26 12:33:43 on dist-test-slave-t7tk"
server_key: "85daadb1df5067380c6771118d0957b6"
server_key_iv: "4ce08c71235a0b52314899d76e7f7cf5"
server_key_version: "encryptionkey@0"
I20260926 12:33:43.358748 14785 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 12:33:43.361390 14799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:43.362776 14785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:33:43.362895 14785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "6e0957a0067c423b997e4e207381e710"
format_stamp: "Formatted at 2026-09-26 12:33:43 on dist-test-slave-t7tk"
server_key: "85daadb1df5067380c6771118d0957b6"
server_key_iv: "4ce08c71235a0b52314899d76e7f7cf5"
server_key_version: "encryptionkey@0"
I20260926 12:33:43.363013 14785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:43.377040 14785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:43.380429 14785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:43.380617 14785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:43.381232 14785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:43.382052 14785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:43.382100 14785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:43.382154 14785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:43.382184 14785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:43.390714 14785 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:35663
I20260926 12:33:43.390733 14912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:35663 every 8 connection(s)
I20260926 12:33:43.391301 14785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:33:43.396461 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 14785
I20260926 12:33:43.396598 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:33:43.396848 10632 external_mini_cluster.cc:1428] Setting key aff0879bf57a4d12264d5b3ba7237d9c
Sep 26 12:33:43 dist-test-slave-t7tk krb5kdc[14679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426023, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.98.2@KRBTEST.COM; defaulting to no policy
I20260926 12:33:43.405670 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:43.393055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:43735 (local address 127.10.98.62:40903)
0926 12:33:43.393208 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:43.393213 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:43.394125 (+ 912us) server_negotiation.cc:407] Connection header received
0926 12:33:43.395415 (+ 1290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:43.395420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:43.395477 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:43.395550 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:43.396792 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.397415 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.398739 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.399035 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.399126 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:43.401532 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:43.401535 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:43.401537 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:43.401562 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:43.403156 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:43.403762 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:43.403765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:43.403766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:43.403844 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:43.404363 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:43.404368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:43.404369 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:43.404580 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:43.404686 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:43.405344 (+ 658us) server_negotiation.cc:299] Negotiation successful
0926 12:33:43.405499 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:33:43.406757 14915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:43.393502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40903 (local address 127.10.98.1:43735)
0926 12:33:43.393973 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:43.394021 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:43.395174 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:43.395740 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:43.395748 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:43.396128 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:43.396648 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:43.396667 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.397540 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.397544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:43.398605 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:43.398615 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.399184 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.399189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:43.399356 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:43.399898 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:43.399915 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:43.401370 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:43.403291 (+ 1921us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:43.403298 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:43.403316 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:43.403598 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:43.403935 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:43.403938 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:43.404049 (+ 111us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:43.404229 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:43.404703 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:43.404711 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:43.404991 (+ 280us) client_negotiation.cc:769] Sending connection context
0926 12:33:43.405239 (+ 248us) client_negotiation.cc:240] Negotiation successful
0926 12:33:43.405782 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":111,"threads_started":1}
I20260926 12:33:43.408159 14913 heartbeater.cc:347] Connected to a master server at 127.10.98.62:40903
I20260926 12:33:43.408463 14913 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.2@KRBTEST.COM" created.
I20260926 12:33:43.408959 14913 heartbeater.cc:511] Master 127.10.98.62:40903 requested a full tablet report, sending...
I20260926 12:33:43.410709 14723 ts_manager.cc:194] Registered new tserver with Master: 6e0957a0067c423b997e4e207381e710 (127.10.98.1:35663)
I20260926 12:33:43.411962 14723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:43735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
I20260926 12:33:43.425684 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:40903
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:43.538614 14918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:43.538915 14918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:43.538961 14918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:43.543277 14918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:43.543347 14918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:43.543429 14918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:33:43.548916 14918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:40903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:43.550040 14918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:43.551112 14918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:43.561843 14923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:43.561864 14926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:43.561892 14924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:43.562114 14918 server_base.cc:1034] running on GCE node
I20260926 12:33:43.562623 14918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:43.563303 14918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:43.564512 14918 hybrid_clock.cc:648] HybridClock initialized: now 1790426023564490 us; error 33 us; skew 500 ppm
Sep 26 12:33:43 dist-test-slave-t7tk krb5kdc[14679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426023, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:43.567533 14918 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:33:43.568231 14918 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:43.568293 14918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:43.568470 14918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:43.570691 14918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "ae8571b55a6e4e0ba89dfc742846112d"
format_stamp: "Formatted at 2026-09-26 12:33:43 on dist-test-slave-t7tk"
server_key: "5b896bc65a1138907fbf2f5aa9f1e5c9"
server_key_iv: "f08b808d34035bfdc97227d951bd560b"
server_key_version: "encryptionkey@0"
I20260926 12:33:43.571220 14918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "ae8571b55a6e4e0ba89dfc742846112d"
format_stamp: "Formatted at 2026-09-26 12:33:43 on dist-test-slave-t7tk"
server_key: "5b896bc65a1138907fbf2f5aa9f1e5c9"
server_key_iv: "f08b808d34035bfdc97227d951bd560b"
server_key_version: "encryptionkey@0"
I20260926 12:33:43.575495 14918 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:33:43.578274 14932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:43.579629 14918 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:33:43.579741 14918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "ae8571b55a6e4e0ba89dfc742846112d"
format_stamp: "Formatted at 2026-09-26 12:33:43 on dist-test-slave-t7tk"
server_key: "5b896bc65a1138907fbf2f5aa9f1e5c9"
server_key_iv: "f08b808d34035bfdc97227d951bd560b"
server_key_version: "encryptionkey@0"
I20260926 12:33:43.579844 14918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:43.599762 14918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:43.603062 14918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:43.603322 14918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:43.604020 14918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:43.604859 14918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:43.604916 14918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:43.604955 14918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:43.604969 14918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:43.613823 14918 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:36859
I20260926 12:33:43.613900 15045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:36859 every 8 connection(s)
I20260926 12:33:43.614445 14918 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:33:43.622277 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 14918
I20260926 12:33:43.622399 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:33:43.622660 10632 external_mini_cluster.cc:1428] Setting key 71a341ec703b12ba5595057083dbcfe3
Sep 26 12:33:43 dist-test-slave-t7tk krb5kdc[14679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426023, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:43.629475 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:43.616353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:40977 (local address 127.10.98.62:40903)
0926 12:33:43.616519 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:43.616525 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:43.617836 (+ 1311us) server_negotiation.cc:407] Connection header received
0926 12:33:43.618735 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:43.618738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:43.618787 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:43.618867 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:43.620171 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.620734 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.621992 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.622231 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.622340 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:43.625253 (+ 2913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:43.625258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:43.625261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:43.625291 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:43.626819 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:43.627498 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:43.627502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:43.627504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:43.627601 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:43.627963 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:43.627966 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:43.627968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:43.628189 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:43.628288 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:43.629156 (+ 868us) server_negotiation.cc:299] Negotiation successful
0926 12:33:43.629297 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 12:33:43.630291 15048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:43.616748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40903 (local address 127.10.98.2:40977)
0926 12:33:43.617650 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:43.617708 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:43.618529 (+ 821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:43.619079 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:43.619087 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:43.619470 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:43.619992 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:43.620007 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.620876 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.620879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:43.621870 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:43.621881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.622445 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.622450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:43.622629 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:43.623270 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:43.623291 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:43.625052 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:43.626982 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:43.626990 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:43.627008 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:43.627363 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:43.627697 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:43.627699 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:43.627701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:43.627868 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:43.628429 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:43.628436 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:43.628849 (+ 413us) client_negotiation.cc:769] Sending connection context
0926 12:33:43.629024 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 12:33:43.629360 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":125,"threads_started":1}
I20260926 12:33:43.631516 15046 heartbeater.cc:347] Connected to a master server at 127.10.98.62:40903
I20260926 12:33:43.631861 15046 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:43.632455 15046 heartbeater.cc:511] Master 127.10.98.62:40903 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.98.3@KRBTEST.COM; defaulting to no policy
I20260926 12:33:43.633741 14723 ts_manager.cc:194] Registered new tserver with Master: ae8571b55a6e4e0ba89dfc742846112d (127.10.98.2:36859)
I20260926 12:33:43.634372 14723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:40977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
I20260926 12:33:43.652733 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:40903
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:43.763839 15051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:43.764139 15051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:43.764185 15051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:43.768388 15051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:43.768466 15051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:43.768548 15051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:33:43.774328 15051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:40903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:43.775549 15051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:43.776636 15051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:43.786605 15059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:43.786605 15056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:43.786724 15051 server_base.cc:1034] running on GCE node
W20260926 12:33:43.786654 15057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:43.787231 15051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:43.787884 15051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:43.789065 15051 hybrid_clock.cc:648] HybridClock initialized: now 1790426023789041 us; error 38 us; skew 500 ppm
Sep 26 12:33:43 dist-test-slave-t7tk krb5kdc[14679](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426023, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:43.791857 15051 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:33:43.792613 15051 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:43.792683 15051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:43.792879 15051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:43.795027 15051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "a4a90e9d2895461db004b5c05d7c4534"
format_stamp: "Formatted at 2026-09-26 12:33:43 on dist-test-slave-t7tk"
server_key: "fc5c989fd82ccf598a0fe4e7043503e1"
server_key_iv: "566764e575c8e0301de1711ac144a54f"
server_key_version: "encryptionkey@0"
I20260926 12:33:43.795583 15051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "a4a90e9d2895461db004b5c05d7c4534"
format_stamp: "Formatted at 2026-09-26 12:33:43 on dist-test-slave-t7tk"
server_key: "fc5c989fd82ccf598a0fe4e7043503e1"
server_key_iv: "566764e575c8e0301de1711ac144a54f"
server_key_version: "encryptionkey@0"
I20260926 12:33:43.799355 15051 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:33:43.801905 15065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:43.803288 15051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:33:43.803416 15051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "a4a90e9d2895461db004b5c05d7c4534"
format_stamp: "Formatted at 2026-09-26 12:33:43 on dist-test-slave-t7tk"
server_key: "fc5c989fd82ccf598a0fe4e7043503e1"
server_key_iv: "566764e575c8e0301de1711ac144a54f"
server_key_version: "encryptionkey@0"
I20260926 12:33:43.803519 15051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:43.821491 15051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:43.824554 15051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:43.824743 15051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:43.825369 15051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:43.826252 15051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:43.826303 15051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:43.826367 15051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:43.826392 15051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:43.834874 15051 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:41879
I20260926 12:33:43.834916 15178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:41879 every 8 connection(s)
I20260926 12:33:43.835469 15051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:33:43.839514 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 15051
I20260926 12:33:43.839640 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:33:43.839955 10632 external_mini_cluster.cc:1428] Setting key d676b2b5f206e573a025cecd2e1f29cb
Sep 26 12:33:43 dist-test-slave-t7tk krb5kdc[14679](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426023, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:43.849121 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:43.837213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:45423 (local address 127.10.98.62:40903)
0926 12:33:43.837343 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:43.837347 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:43.838195 (+ 848us) server_negotiation.cc:407] Connection header received
0926 12:33:43.839552 (+ 1357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:43.839556 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:43.839616 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:43.839691 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:43.841214 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.841965 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.843300 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.843533 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.843621 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:43.846021 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:43.846025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:43.846028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:43.846056 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:43.847140 (+ 1084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:43.847629 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:43.847633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:43.847635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:43.847713 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:43.848029 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:43.848032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:43.848034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:43.848211 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:43.848295 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:43.848811 (+ 516us) server_negotiation.cc:299] Negotiation successful
0926 12:33:43.848936 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:33:43.849944 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:43.837597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40903 (local address 127.10.98.3:45423)
0926 12:33:43.838052 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:43.838094 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:43.839334 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:43.840043 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:43.840053 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:43.840504 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:43.841038 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:43.841057 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.842094 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.842098 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:43.843185 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:43.843195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.843675 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.843679 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:43.843835 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:43.844363 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:43.844381 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:43.845855 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:43.847244 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:43.847250 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:43.847266 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:43.847524 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:43.847788 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:43.847792 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:43.847794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:43.847936 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:43.848304 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:43.848311 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:43.848570 (+ 259us) client_negotiation.cc:769] Sending connection context
0926 12:33:43.848732 (+ 162us) client_negotiation.cc:240] Negotiation successful
0926 12:33:43.849076 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":109,"threads_started":1}
I20260926 12:33:43.851061 15179 heartbeater.cc:347] Connected to a master server at 127.10.98.62:40903
I20260926 12:33:43.851289 15179 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:43.851732 15179 heartbeater.cc:511] Master 127.10.98.62:40903 requested a full tablet report, sending...
I20260926 12:33:43.852871 14723 ts_manager.cc:194] Registered new tserver with Master: a4a90e9d2895461db004b5c05d7c4534 (127.10.98.3:41879)
I20260926 12:33:43.853534 14723 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:45423
I20260926 12:33:43.853945 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:33:43.863250 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:43.856171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34674 (local address 127.10.98.62:40903)
0926 12:33:43.856312 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:43.856317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:43.856461 (+ 144us) server_negotiation.cc:407] Connection header received
0926 12:33:43.856556 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:43.856559 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:43.856595 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:43.856666 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:43.857441 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.857950 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.858826 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.859052 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.859134 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:43.860471 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:43.860474 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:43.860477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:43.860499 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:43.861755 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:43.862117 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:43.862119 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:43.862121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:43.862188 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:43.862415 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:43.862417 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:43.862418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:43.862621 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:43.862737 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:43.862983 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 12:33:43.863095 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:33:43.866410 14723 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:33:43.869516 14723 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:33:43.884033 15190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:43.877843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:35663 (local address 127.0.0.1:37636)
0926 12:33:43.878746 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:43.878798 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:43.878965 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:43.879728 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:43.879731 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:43.879749 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:43.879879 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:43.879886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.881375 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.881379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:43.882499 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:43.882508 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.883428 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.883431 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:43.883586 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:43.883629 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 12:33:43.883745 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 12:33:43.883865 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"mutex_wait_us":109,"thread_start_us":90,"threads_started":1}
I20260926 12:33:43.884071 15193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:43.878122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37636 (local address 127.10.98.1:35663)
0926 12:33:43.878530 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:43.878539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:43.878888 (+ 349us) server_negotiation.cc:407] Connection header received
0926 12:33:43.879119 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:43.879126 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:43.879392 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:43.879530 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:43.879991 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.881201 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.882615 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.883318 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.883427 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:43.883587 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:43.883793 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 12:33:43.883905 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":115,"threads_started":1}
I20260926 12:33:43.884570 15192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:43.877883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44870 (local address 127.10.98.3:41879)
0926 12:33:43.878694 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:43.878704 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:43.879161 (+ 457us) server_negotiation.cc:407] Connection header received
0926 12:33:43.879284 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:43.879289 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:43.879430 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:43.879551 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:43.879941 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.881198 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.882780 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.883630 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.883713 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:43.883850 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:43.884343 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 12:33:43.884435 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":99,"threads_started":1}
I20260926 12:33:43.884267 15191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:43.877725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:41879 (local address 127.0.0.1:44870)
0926 12:33:43.879071 (+ 1346us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:43.879084 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:43.879165 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:43.879571 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:43.879577 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:43.879599 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:43.879819 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:43.879828 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.881321 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.881324 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:43.882664 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:43.882674 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.883909 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.883912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:43.883998 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:43.884018 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 12:33:43.884083 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 12:33:43.884148 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":72,"spinlock_wait_cycles":768,"thread_start_us":86,"threads_started":1}
I20260926 12:33:43.885043 15194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:43.878508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:36859 (local address 127.0.0.1:53646)
0926 12:33:43.878785 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:43.878799 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:43.878925 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:43.879626 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:43.879629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:43.879653 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:43.879881 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:43.879888 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.882476 (+ 2588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.882479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:43.883570 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:43.883581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.884695 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.884700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:43.884784 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:43.884806 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:33:43.884852 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 12:33:43.884919 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"mutex_wait_us":122,"spinlock_wait_cycles":3200,"thread_start_us":88,"threads_started":1}
I20260926 12:33:43.885749 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:43.878833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53646 (local address 127.10.98.2:36859)
0926 12:33:43.879245 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:43.879257 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:43.879276 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:33:43.879336 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:43.879340 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:43.879493 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:43.879755 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:43.881195 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.882326 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.883717 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:43.884567 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:43.885309 (+ 742us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:43.885466 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:43.885520 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 12:33:43.885607 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":149,"threads_started":1}
I20260926 12:33:43.887619 14847 tablet_service.cc:1467] Processing CreateTablet for tablet a4e8a5019c06493a8c5b6d1c44f959c9 (DEFAULT_TABLE table=test-table [id=cc853a9edd2a40dc8d625adad66cbd21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:43.888345 14980 tablet_service.cc:1467] Processing CreateTablet for tablet a4e8a5019c06493a8c5b6d1c44f959c9 (DEFAULT_TABLE table=test-table [id=cc853a9edd2a40dc8d625adad66cbd21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:43.888739 14847 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4e8a5019c06493a8c5b6d1c44f959c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:43.889299 14980 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4e8a5019c06493a8c5b6d1c44f959c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:43.890106 15113 tablet_service.cc:1467] Processing CreateTablet for tablet a4e8a5019c06493a8c5b6d1c44f959c9 (DEFAULT_TABLE table=test-table [id=cc853a9edd2a40dc8d625adad66cbd21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:43.891084 15113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4e8a5019c06493a8c5b6d1c44f959c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:43.895315 15196 tablet_bootstrap.cc:492] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d: Bootstrap starting.
I20260926 12:33:43.896675 15197 tablet_bootstrap.cc:492] T a4e8a5019c06493a8c5b6d1c44f959c9 P 6e0957a0067c423b997e4e207381e710: Bootstrap starting.
I20260926 12:33:43.897616 15198 tablet_bootstrap.cc:492] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534: Bootstrap starting.
I20260926 12:33:43.898226 15196 tablet_bootstrap.cc:654] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:43.898800 15197 tablet_bootstrap.cc:654] T a4e8a5019c06493a8c5b6d1c44f959c9 P 6e0957a0067c423b997e4e207381e710: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:43.899106 15196 log.cc:824] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:43.899451 15198 tablet_bootstrap.cc:654] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:43.900007 15197 log.cc:824] T a4e8a5019c06493a8c5b6d1c44f959c9 P 6e0957a0067c423b997e4e207381e710: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:43.900625 15198 log.cc:824] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:43.901463 15196 tablet_bootstrap.cc:492] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d: No bootstrap required, opened a new log
I20260926 12:33:43.901669 15196 ts_tablet_manager.cc:1397] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 12:33:43.902065 15197 tablet_bootstrap.cc:492] T a4e8a5019c06493a8c5b6d1c44f959c9 P 6e0957a0067c423b997e4e207381e710: No bootstrap required, opened a new log
I20260926 12:33:43.902251 15197 ts_tablet_manager.cc:1397] T a4e8a5019c06493a8c5b6d1c44f959c9 P 6e0957a0067c423b997e4e207381e710: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:33:43.902505 15198 tablet_bootstrap.cc:492] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534: No bootstrap required, opened a new log
I20260926 12:33:43.902675 15198 ts_tablet_manager.cc:1397] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 12:33:43.905467 15198 raft_consensus.cc:348] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a90e9d2895461db004b5c05d7c4534" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 41879 } } peers { permanent_uuid: "6e0957a0067c423b997e4e207381e710" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35663 } } peers { permanent_uuid: "ae8571b55a6e4e0ba89dfc742846112d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 36859 } }
I20260926 12:33:43.905707 15198 raft_consensus.cc:374] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:43.905514 15196 raft_consensus.cc:348] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a90e9d2895461db004b5c05d7c4534" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 41879 } } peers { permanent_uuid: "6e0957a0067c423b997e4e207381e710" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35663 } } peers { permanent_uuid: "ae8571b55a6e4e0ba89dfc742846112d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 36859 } }
I20260926 12:33:43.905759 15198 raft_consensus.cc:729] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4a90e9d2895461db004b5c05d7c4534, State: Initialized, Role: FOLLOWER
I20260926 12:33:43.905781 15196 raft_consensus.cc:374] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:43.905830 15196 raft_consensus.cc:729] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8571b55a6e4e0ba89dfc742846112d, State: Initialized, Role: FOLLOWER
I20260926 12:33:43.905903 15197 raft_consensus.cc:348] T a4e8a5019c06493a8c5b6d1c44f959c9 P 6e0957a0067c423b997e4e207381e710 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a90e9d2895461db004b5c05d7c4534" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 41879 } } peers { permanent_uuid: "6e0957a0067c423b997e4e207381e710" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35663 } } peers { permanent_uuid: "ae8571b55a6e4e0ba89dfc742846112d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 36859 } }
I20260926 12:33:43.906131 15197 raft_consensus.cc:374] T a4e8a5019c06493a8c5b6d1c44f959c9 P 6e0957a0067c423b997e4e207381e710 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:43.906184 15197 raft_consensus.cc:729] T a4e8a5019c06493a8c5b6d1c44f959c9 P 6e0957a0067c423b997e4e207381e710 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e0957a0067c423b997e4e207381e710, State: Initialized, Role: FOLLOWER
I20260926 12:33:43.906279 15198 consensus_queue.cc:260] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a90e9d2895461db004b5c05d7c4534" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 41879 } } peers { permanent_uuid: "6e0957a0067c423b997e4e207381e710" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35663 } } peers { permanent_uuid: "ae8571b55a6e4e0ba89dfc742846112d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 36859 } }
I20260926 12:33:43.906334 15196 consensus_queue.cc:260] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a90e9d2895461db004b5c05d7c4534" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 41879 } } peers { permanent_uuid: "6e0957a0067c423b997e4e207381e710" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35663 } } peers { permanent_uuid: "ae8571b55a6e4e0ba89dfc742846112d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 36859 } }
I20260926 12:33:43.906543 15197 consensus_queue.cc:260] T a4e8a5019c06493a8c5b6d1c44f959c9 P 6e0957a0067c423b997e4e207381e710 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a90e9d2895461db004b5c05d7c4534" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 41879 } } peers { permanent_uuid: "6e0957a0067c423b997e4e207381e710" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35663 } } peers { permanent_uuid: "ae8571b55a6e4e0ba89dfc742846112d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 36859 } }
I20260926 12:33:43.907001 15179 heartbeater.cc:503] Master 127.10.98.62:40903 was elected leader, sending a full tablet report...
I20260926 12:33:43.907286 15196 ts_tablet_manager.cc:1428] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:33:43.907280 15198 ts_tablet_manager.cc:1428] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:33:43.907624 15046 heartbeater.cc:503] Master 127.10.98.62:40903 was elected leader, sending a full tablet report...
I20260926 12:33:43.907895 14913 heartbeater.cc:503] Master 127.10.98.62:40903 was elected leader, sending a full tablet report...
I20260926 12:33:43.908003 15197 ts_tablet_manager.cc:1428] T a4e8a5019c06493a8c5b6d1c44f959c9 P 6e0957a0067c423b997e4e207381e710: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:33:44.078687 15202 raft_consensus.cc:482] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:33:44.079015 15202 raft_consensus.cc:504] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a90e9d2895461db004b5c05d7c4534" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 41879 } } peers { permanent_uuid: "6e0957a0067c423b997e4e207381e710" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35663 } } peers { permanent_uuid: "ae8571b55a6e4e0ba89dfc742846112d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 36859 } }
I20260926 12:33:44.080248 15202 leader_election.cc:288] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e0957a0067c423b997e4e207381e710 (127.10.98.1:35663), ae8571b55a6e4e0ba89dfc742846112d (127.10.98.2:36859)
I20260926 12:33:44.085271 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:44.080593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:35663 (local address 127.10.98.3:39323)
0926 12:33:44.080719 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:44.080734 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:44.080830 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:44.081157 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:44.081160 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:44.081179 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:44.081358 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:44.081365 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:44.082337 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:44.082340 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:44.083671 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:44.083683 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:44.084703 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:44.084707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:44.084915 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:44.084933 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 12:33:44.084997 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 12:33:44.085081 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":27}
I20260926 12:33:44.085285 15193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:44.080630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:39323 (local address 127.10.98.1:35663)
0926 12:33:44.080753 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:44.080757 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:44.080788 (+ 31us) server_negotiation.cc:407] Connection header received
0926 12:33:44.080957 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:44.080961 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:44.081021 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:44.081140 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:44.081468 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:44.082221 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:44.083912 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:44.084556 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:44.084656 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:44.084722 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:44.085045 (+ 323us) server_negotiation.cc:299] Negotiation successful
0926 12:33:44.085155 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:33:44.085304 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:44.080732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:49873 (local address 127.10.98.2:36859)
0926 12:33:44.080856 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:44.080861 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:44.081276 (+ 415us) server_negotiation.cc:407] Connection header received
0926 12:33:44.081360 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:44.081362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:44.081401 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:44.081477 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:44.081683 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:44.082373 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:44.083863 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:44.084544 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:44.084656 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:44.084737 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:44.085045 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 12:33:44.085165 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 12:33:44.085276 15205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:44.080597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:36859 (local address 127.10.98.3:49873)
0926 12:33:44.081177 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:44.081190 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:44.081279 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:44.081491 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:44.081493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:44.081508 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:44.081591 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:44.081595 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:44.082468 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:44.082471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:44.083732 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:44.083739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:44.084703 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:44.084707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:44.084869 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:44.084898 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 12:33:44.084987 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 12:33:44.085081 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":181,"thread_start_us":207,"threads_started":1}
I20260926 12:33:44.086155 14867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e8a5019c06493a8c5b6d1c44f959c9" candidate_uuid: "a4a90e9d2895461db004b5c05d7c4534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e0957a0067c423b997e4e207381e710" is_pre_election: true
I20260926 12:33:44.086103 15000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e8a5019c06493a8c5b6d1c44f959c9" candidate_uuid: "a4a90e9d2895461db004b5c05d7c4534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8571b55a6e4e0ba89dfc742846112d" is_pre_election: true
W20260926 12:33:44.086305 15180 tablet.cc:2369] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:33:44.086477 15000 raft_consensus.cc:2454] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a90e9d2895461db004b5c05d7c4534 in term 0.
I20260926 12:33:44.086476 14867 raft_consensus.cc:2454] T a4e8a5019c06493a8c5b6d1c44f959c9 P 6e0957a0067c423b997e4e207381e710 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4a90e9d2895461db004b5c05d7c4534 in term 0.
I20260926 12:33:44.086989 15067 leader_election.cc:302] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e0957a0067c423b997e4e207381e710, a4a90e9d2895461db004b5c05d7c4534; no voters:
I20260926 12:33:44.087262 15202 raft_consensus.cc:2781] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:33:44.087349 15202 raft_consensus.cc:482] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:33:44.087400 15202 raft_consensus.cc:3037] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:44.088431 15202 raft_consensus.cc:504] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a90e9d2895461db004b5c05d7c4534" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 41879 } } peers { permanent_uuid: "6e0957a0067c423b997e4e207381e710" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35663 } } peers { permanent_uuid: "ae8571b55a6e4e0ba89dfc742846112d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 36859 } }
I20260926 12:33:44.088764 15202 leader_election.cc:288] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 [CANDIDATE]: Term 1 election: Requested vote from peers 6e0957a0067c423b997e4e207381e710 (127.10.98.1:35663), ae8571b55a6e4e0ba89dfc742846112d (127.10.98.2:36859)
I20260926 12:33:44.089164 15000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e8a5019c06493a8c5b6d1c44f959c9" candidate_uuid: "a4a90e9d2895461db004b5c05d7c4534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8571b55a6e4e0ba89dfc742846112d"
I20260926 12:33:44.089279 14867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e8a5019c06493a8c5b6d1c44f959c9" candidate_uuid: "a4a90e9d2895461db004b5c05d7c4534" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e0957a0067c423b997e4e207381e710"
I20260926 12:33:44.089305 15000 raft_consensus.cc:3037] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:44.089407 14867 raft_consensus.cc:3037] T a4e8a5019c06493a8c5b6d1c44f959c9 P 6e0957a0067c423b997e4e207381e710 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:44.090384 14867 raft_consensus.cc:2454] T a4e8a5019c06493a8c5b6d1c44f959c9 P 6e0957a0067c423b997e4e207381e710 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a90e9d2895461db004b5c05d7c4534 in term 1.
I20260926 12:33:44.090399 15000 raft_consensus.cc:2454] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4a90e9d2895461db004b5c05d7c4534 in term 1.
I20260926 12:33:44.090668 15067 leader_election.cc:302] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e0957a0067c423b997e4e207381e710, a4a90e9d2895461db004b5c05d7c4534; no voters:
I20260926 12:33:44.090833 15202 raft_consensus.cc:2781] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:44.091037 15202 raft_consensus.cc:686] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 [term 1 LEADER]: Becoming Leader. State: Replica: a4a90e9d2895461db004b5c05d7c4534, State: Running, Role: LEADER
I20260926 12:33:44.091349 15202 consensus_queue.cc:237] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a90e9d2895461db004b5c05d7c4534" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 41879 } } peers { permanent_uuid: "6e0957a0067c423b997e4e207381e710" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35663 } } peers { permanent_uuid: "ae8571b55a6e4e0ba89dfc742846112d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 36859 } }
I20260926 12:33:44.094736 14723 catalog_manager.cc:5495] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 reported cstate change: term changed from 0 to 1, leader changed from <none> to a4a90e9d2895461db004b5c05d7c4534 (127.10.98.3). New cstate: current_term: 1 leader_uuid: "a4a90e9d2895461db004b5c05d7c4534" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a4a90e9d2895461db004b5c05d7c4534" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 41879 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e0957a0067c423b997e4e207381e710" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35663 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae8571b55a6e4e0ba89dfc742846112d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 36859 } health_report { overall_health: UNKNOWN } } }
W20260926 12:33:44.115556 15047 tablet.cc:2369] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:33:44.123126 15192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:44.118771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44872 (local address 127.10.98.3:41879)
0926 12:33:44.118924 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:44.118930 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:44.118946 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:33:44.119110 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:44.119114 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:44.119180 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:44.119288 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:44.119523 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:44.120294 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:44.121393 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:44.121744 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:44.121839 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:44.122164 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:44.122687 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:44.122802 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:44.122897 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 12:33:44.122957 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 12:33:44.129917 15000 raft_consensus.cc:1261] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d [term 1 FOLLOWER]: Refusing update from remote peer a4a90e9d2895461db004b5c05d7c4534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:44.129947 14867 raft_consensus.cc:1261] T a4e8a5019c06493a8c5b6d1c44f959c9 P 6e0957a0067c423b997e4e207381e710 [term 1 FOLLOWER]: Refusing update from remote peer a4a90e9d2895461db004b5c05d7c4534: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:44.130667 15206 consensus_queue.cc:1035] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e0957a0067c423b997e4e207381e710" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35663 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:44.130867 15202 consensus_queue.cc:1035] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8571b55a6e4e0ba89dfc742846112d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 36859 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:44.138127 15209 mvcc.cc:205] Tried to move back new op lower bound from 7333584994827534336 to 7333584994680766464. Current Snapshot: MvccSnapshot[applied={T|T < 7333584994827534336}]
I20260926 12:33:44.139019 15210 mvcc.cc:205] Tried to move back new op lower bound from 7333584994827534336 to 7333584994680766464. Current Snapshot: MvccSnapshot[applied={T|T < 7333584994827534336}]
I20260926 12:33:44.148361 14723 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:44.148542 14723 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:44.151214 14723 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a6e361b1ac2946f0adbf29b8368962d2: Sending DeleteTablet for 3 replicas of tablet a4e8a5019c06493a8c5b6d1c44f959c9
I20260926 12:33:44.152181 14980 tablet_service.cc:1514] Processing DeleteTablet for tablet a4e8a5019c06493a8c5b6d1c44f959c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:44 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:53646
I20260926 12:33:44.152336 15113 tablet_service.cc:1514] Processing DeleteTablet for tablet a4e8a5019c06493a8c5b6d1c44f959c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:44 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:44870
I20260926 12:33:44.152752 14847 tablet_service.cc:1514] Processing DeleteTablet for tablet a4e8a5019c06493a8c5b6d1c44f959c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:44 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:37636
I20260926 12:33:44.152844 15219 tablet_replica.cc:330] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534: stopping tablet replica
I20260926 12:33:44.153264 15220 tablet_replica.cc:330] T a4e8a5019c06493a8c5b6d1c44f959c9 P 6e0957a0067c423b997e4e207381e710: stopping tablet replica
I20260926 12:33:44.153298 15219 raft_consensus.cc:2229] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:33:44.153424 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 14785
I20260926 12:33:44.153565 15220 raft_consensus.cc:2229] T a4e8a5019c06493a8c5b6d1c44f959c9 P 6e0957a0067c423b997e4e207381e710 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:44.153921 15219 raft_consensus.cc:2258] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:44.155660 15218 tablet_replica.cc:330] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d: stopping tablet replica
I20260926 12:33:44.155974 15218 raft_consensus.cc:2229] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:44.156301 15218 raft_consensus.cc:2258] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:44.156692 15219 ts_tablet_manager.cc:1905] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:44.158951 15218 ts_tablet_manager.cc:1905] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:44.159778 15219 ts_tablet_manager.cc:1918] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:33:44.159963 15219 log.cc:1196] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/a4e8a5019c06493a8c5b6d1c44f959c9
I20260926 12:33:44.160320 15219 ts_tablet_manager.cc:1939] T a4e8a5019c06493a8c5b6d1c44f959c9 P a4a90e9d2895461db004b5c05d7c4534: Deleting consensus metadata
I20260926 12:33:44.161461 15218 ts_tablet_manager.cc:1918] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:33:44.161553 15218 log.cc:1196] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/a4e8a5019c06493a8c5b6d1c44f959c9
I20260926 12:33:44.161823 15218 ts_tablet_manager.cc:1939] T a4e8a5019c06493a8c5b6d1c44f959c9 P ae8571b55a6e4e0ba89dfc742846112d: Deleting consensus metadata
I20260926 12:33:44.163203 14710 catalog_manager.cc:4841] TS ae8571b55a6e4e0ba89dfc742846112d (127.10.98.2:36859): tablet a4e8a5019c06493a8c5b6d1c44f959c9 (table test-table [id=cc853a9edd2a40dc8d625adad66cbd21]) successfully deleted
I20260926 12:33:44.163411 14710 catalog_manager.cc:4841] TS a4a90e9d2895461db004b5c05d7c4534 (127.10.98.3:41879): tablet a4e8a5019c06493a8c5b6d1c44f959c9 (table test-table [id=cc853a9edd2a40dc8d625adad66cbd21]) successfully deleted
W20260926 12:33:44.163753 14708 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.98.1:35663 (error 108)
I20260926 12:33:44.163959 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 14918
I20260926 12:33:44.165167 15191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:44.164858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:35663 (local address 127.0.0.1:37642)
0926 12:33:44.164984 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:44.165054 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.1:35663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260926 12:33:44.165395 14708 catalog_manager.cc:4584] TS 6e0957a0067c423b997e4e207381e710 (127.10.98.1:35663): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4e8a5019c06493a8c5b6d1c44f959c9: Network error: Client connection negotiation failed: client connection to 127.10.98.1:35663: connect: Connection refused (error 111)
I20260926 12:33:44.170408 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 15051
I20260926 12:33:44.175150 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 14692
2026-09-26T12:33:44Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2347 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T12:33:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:33:44Z Disabled control of system clock
I20260926 12:33:44.203652 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:46253
--webserver_interface=127.10.98.62
--webserver_port=0
--builtin_ntp_servers=127.10.98.20:37489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:46253
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 12:33:44.317549 15225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:44.317859 15225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:44.317901 15225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:44.322052 15225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:44.322130 15225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:44.322147 15225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:44.322170 15225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:44.322188 15225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:33:44.327219 15225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:37489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:46253
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:46253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:44.328321 15225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:44.329790 15225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:44.338222 15230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:44.338292 15233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:44.338269 15231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:44.338642 15225 server_base.cc:1034] running on GCE node
I20260926 12:33:44.339262 15225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:44.340428 15225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:44.341612 15225 hybrid_clock.cc:648] HybridClock initialized: now 1790426024341580 us; error 48 us; skew 500 ppm
I20260926 12:33:44.343726 15225 webserver.cc:460] Webserver started at http://127.10.98.62:45897/ using document root <none> and password file <none>
I20260926 12:33:44.344323 15225 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:44.344394 15225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:44.344591 15225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:44.346740 15225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "adc60cff0e6444a2ae38d9724e8bbbff"
format_stamp: "Formatted at 2026-09-26 12:33:44 on dist-test-slave-t7tk"
server_key: "b9854ae7d81029234166816fa7c75562"
server_key_iv: "8be48e8bf834258d7e8b79baa90840cb"
server_key_version: "encryptionkey@0"
I20260926 12:33:44.347426 15225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "adc60cff0e6444a2ae38d9724e8bbbff"
format_stamp: "Formatted at 2026-09-26 12:33:44 on dist-test-slave-t7tk"
server_key: "b9854ae7d81029234166816fa7c75562"
server_key_iv: "8be48e8bf834258d7e8b79baa90840cb"
server_key_version: "encryptionkey@0"
I20260926 12:33:44.351294 15225 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:33:44.353852 15239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:44.355233 15225 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:33:44.355363 15225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "adc60cff0e6444a2ae38d9724e8bbbff"
format_stamp: "Formatted at 2026-09-26 12:33:44 on dist-test-slave-t7tk"
server_key: "b9854ae7d81029234166816fa7c75562"
server_key_iv: "8be48e8bf834258d7e8b79baa90840cb"
server_key_version: "encryptionkey@0"
I20260926 12:33:44.355474 15225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:44.360110 15225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:44.360733 15225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:44.360898 15225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:44.367410 15225 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:46253
I20260926 12:33:44.367430 15291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:46253 every 8 connection(s)
I20260926 12:33:44.368407 15225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:33:44.369468 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 15225
I20260926 12:33:44.369580 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:33:44.369908 10632 external_mini_cluster.cc:1428] Setting key 93af60cdf23a03096b4cab458ded7f48
I20260926 12:33:44.371940 15292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:44.376231 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:44.371543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50192 (local address 127.10.98.62:46253)
0926 12:33:44.372758 (+ 1215us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:44.372775 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:44.372810 (+ 35us) server_negotiation.cc:407] Connection header received
0926 12:33:44.373503 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:44.373521 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:44.373894 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:44.374219 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:44.374999 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:44.375004 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:44.375016 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:33:44.375026 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:44.375080 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:33:44.375096 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:44.375235 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:44.375306 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 12:33:44.375544 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":851,"thread_start_us":686,"threads_started":1}
I20260926 12:33:44.378080 15292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff: Bootstrap starting.
I20260926 12:33:44.380330 15292 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:44.381070 15292 log.cc:824] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:44.382993 15292 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff: No bootstrap required, opened a new log
I20260926 12:33:44.386776 15292 raft_consensus.cc:348] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc60cff0e6444a2ae38d9724e8bbbff" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 46253 } }
I20260926 12:33:44.387019 15292 raft_consensus.cc:374] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:44.387058 15292 raft_consensus.cc:729] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc60cff0e6444a2ae38d9724e8bbbff, State: Initialized, Role: FOLLOWER
I20260926 12:33:44.387473 15292 consensus_queue.cc:260] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc60cff0e6444a2ae38d9724e8bbbff" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 46253 } }
I20260926 12:33:44.387609 15292 raft_consensus.cc:388] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:33:44.387666 15292 raft_consensus.cc:482] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:33:44.387745 15292 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:44.388656 15292 raft_consensus.cc:504] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc60cff0e6444a2ae38d9724e8bbbff" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 46253 } }
I20260926 12:33:44.389019 15292 leader_election.cc:302] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adc60cff0e6444a2ae38d9724e8bbbff; no voters:
I20260926 12:33:44.389355 15292 leader_election.cc:288] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:33:44.389490 15297 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:44.389707 15297 raft_consensus.cc:686] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff [term 1 LEADER]: Becoming Leader. State: Replica: adc60cff0e6444a2ae38d9724e8bbbff, State: Running, Role: LEADER
I20260926 12:33:44.390061 15297 consensus_queue.cc:237] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc60cff0e6444a2ae38d9724e8bbbff" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 46253 } }
I20260926 12:33:44.390749 15292 sys_catalog.cc:564] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:44.391726 15298 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adc60cff0e6444a2ae38d9724e8bbbff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc60cff0e6444a2ae38d9724e8bbbff" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 46253 } } }
I20260926 12:33:44.391871 15298 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:44.392148 15299 sys_catalog.cc:455] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff [sys.catalog]: SysCatalogTable state changed. Reason: New leader adc60cff0e6444a2ae38d9724e8bbbff. Latest consensus state: current_term: 1 leader_uuid: "adc60cff0e6444a2ae38d9724e8bbbff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adc60cff0e6444a2ae38d9724e8bbbff" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 46253 } } }
I20260926 12:33:44.392257 15299 sys_catalog.cc:458] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:44.392575 15306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:33:44.395736 15306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:33:44.401484 15306 catalog_manager.cc:1347] Generated new cluster ID: 3a9ae4caadd94f349b1ef81b8611ce65
I20260926 12:33:44.401548 15306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:33:44.413283 15306 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:33:44.413606 15306 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:33:44.421828 15306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff: Generated new TSK 0
I20260926 12:33:44.422680 15306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:33:44.431877 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:46253
--builtin_ntp_servers=127.10.98.20:37489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 12:33:44.542672 15316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:44.542975 15316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:44.543031 15316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:44.547151 15316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:44.547219 15316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:44.547302 15316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:33:44.552284 15316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:37489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:46253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:44.553484 15316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:44.554876 15316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:44.564750 15322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:44.564846 15321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:44.564888 15324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:44.564987 15316 server_base.cc:1034] running on GCE node
I20260926 12:33:44.565526 15316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:44.566143 15316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:44.567348 15316 hybrid_clock.cc:648] HybridClock initialized: now 1790426024567331 us; error 29 us; skew 500 ppm
I20260926 12:33:44.569383 15316 webserver.cc:460] Webserver started at http://127.10.98.1:42155/ using document root <none> and password file <none>
I20260926 12:33:44.569926 15316 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:44.569985 15316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:44.570158 15316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:44.572150 15316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "6a98514e15b3441a884293f7094353d3"
format_stamp: "Formatted at 2026-09-26 12:33:44 on dist-test-slave-t7tk"
server_key: "8acb7dd811435c269404b292246810e9"
server_key_iv: "846f169bbcffaa6ced9265383714e7b3"
server_key_version: "encryptionkey@0"
I20260926 12:33:44.572652 15316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "6a98514e15b3441a884293f7094353d3"
format_stamp: "Formatted at 2026-09-26 12:33:44 on dist-test-slave-t7tk"
server_key: "8acb7dd811435c269404b292246810e9"
server_key_iv: "846f169bbcffaa6ced9265383714e7b3"
server_key_version: "encryptionkey@0"
I20260926 12:33:44.576373 15316 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:33:44.579242 15330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:44.580607 15316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:33:44.580713 15316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "6a98514e15b3441a884293f7094353d3"
format_stamp: "Formatted at 2026-09-26 12:33:44 on dist-test-slave-t7tk"
server_key: "8acb7dd811435c269404b292246810e9"
server_key_iv: "846f169bbcffaa6ced9265383714e7b3"
server_key_version: "encryptionkey@0"
I20260926 12:33:44.580842 15316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:44.585479 15316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:44.586095 15316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:44.586254 15316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:44.586776 15316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:44.587673 15316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:44.587718 15316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:44.587755 15316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:44.587781 15316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:44.596061 15316 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:34997
I20260926 12:33:44.596096 15443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:34997 every 8 connection(s)
I20260926 12:33:44.596984 15316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:33:44.598212 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 15316
I20260926 12:33:44.598327 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:33:44.598587 10632 external_mini_cluster.cc:1428] Setting key a0e157f23b69760cbe2e98b80e423ac3
I20260926 12:33:44.602200 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:46253
--builtin_ntp_servers=127.10.98.20:37489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 12:33:44.603823 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:44.598960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:38511 (local address 127.10.98.62:46253)
0926 12:33:44.599110 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:44.599115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:44.600384 (+ 1269us) server_negotiation.cc:407] Connection header received
0926 12:33:44.601960 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:44.601963 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:44.601998 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:44.602072 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:44.603332 (+ 1260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:44.603335 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:44.603337 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:33:44.603349 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:44.603379 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:33:44.603395 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:44.603496 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:44.603643 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 12:33:44.603687 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:33:44.604799 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:44.599420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:46253 (local address 127.10.98.1:38511)
0926 12:33:44.600225 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:44.600269 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:44.601720 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:44.602269 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:44.602279 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:44.602416 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:44.602993 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:33:44.602994 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:44.603052 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:33:44.603143 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:44.603492 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:44.603501 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:44.603508 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 12:33:44.603672 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 12:33:44.603874 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":113,"threads_started":1}
I20260926 12:33:44.606091 15444 heartbeater.cc:347] Connected to a master server at 127.10.98.62:46253
I20260926 12:33:44.606333 15444 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:44.606750 15444 heartbeater.cc:511] Master 127.10.98.62:46253 requested a full tablet report, sending...
I20260926 12:33:44.608295 15256 ts_manager.cc:194] Registered new tserver with Master: 6a98514e15b3441a884293f7094353d3 (127.10.98.1:34997)
W20260926 12:33:44.712884 15447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:44.713179 15447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:44.713238 15447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:44.717249 15447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:44.717316 15447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:44.717392 15447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:33:44.722230 15447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:37489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:46253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:44.723344 15447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:44.724702 15447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:44.734396 15453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:44.734368 15452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:44.734380 15455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:44.734584 15447 server_base.cc:1034] running on GCE node
I20260926 12:33:44.735297 15447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:44.735958 15447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:44.737161 15447 hybrid_clock.cc:648] HybridClock initialized: now 1790426024737138 us; error 33 us; skew 500 ppm
I20260926 12:33:44.739060 15447 webserver.cc:460] Webserver started at http://127.10.98.2:41305/ using document root <none> and password file <none>
I20260926 12:33:44.739607 15447 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:44.739668 15447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:44.739857 15447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:44.741966 15447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "4426404c20d8454fb3b25198453f6431"
format_stamp: "Formatted at 2026-09-26 12:33:44 on dist-test-slave-t7tk"
server_key: "737c12995ec77177001bfb3b88a72753"
server_key_iv: "7f88dc8d9407b9d4ae207a6720a07cbe"
server_key_version: "encryptionkey@0"
I20260926 12:33:44.742481 15447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "4426404c20d8454fb3b25198453f6431"
format_stamp: "Formatted at 2026-09-26 12:33:44 on dist-test-slave-t7tk"
server_key: "737c12995ec77177001bfb3b88a72753"
server_key_iv: "7f88dc8d9407b9d4ae207a6720a07cbe"
server_key_version: "encryptionkey@0"
I20260926 12:33:44.746282 15447 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:33:44.748898 15461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:44.750350 15447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:33:44.750470 15447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "4426404c20d8454fb3b25198453f6431"
format_stamp: "Formatted at 2026-09-26 12:33:44 on dist-test-slave-t7tk"
server_key: "737c12995ec77177001bfb3b88a72753"
server_key_iv: "7f88dc8d9407b9d4ae207a6720a07cbe"
server_key_version: "encryptionkey@0"
I20260926 12:33:44.750597 15447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:44.755183 15447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:44.755870 15447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:44.756026 15447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:44.756547 15447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:44.757500 15447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:44.757545 15447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:44.757582 15447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:44.757596 15447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:44.766098 15447 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:35867
I20260926 12:33:44.766120 15574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:35867 every 8 connection(s)
I20260926 12:33:44.766983 15447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:33:44.768532 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 15447
I20260926 12:33:44.768643 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:33:44.768953 10632 external_mini_cluster.cc:1428] Setting key 595638b374ed5b5d2a31d111a28d0d79
I20260926 12:33:44.771154 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:46253
--builtin_ntp_servers=127.10.98.20:37489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 12:33:44.773646 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:44.768857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:48253 (local address 127.10.98.62:46253)
0926 12:33:44.769015 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:44.769020 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:44.770189 (+ 1169us) server_negotiation.cc:407] Connection header received
0926 12:33:44.771817 (+ 1628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:44.771820 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:44.771863 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:44.771938 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:44.773204 (+ 1266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:44.773207 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:44.773209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:33:44.773221 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:44.773240 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:33:44.773248 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:44.773322 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:44.773474 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 12:33:44.773518 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 12:33:44.774704 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:44.769269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:46253 (local address 127.10.98.2:48253)
0926 12:33:44.770053 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:44.770095 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:44.771553 (+ 1458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:44.772148 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:44.772158 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:44.772275 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:44.772923 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:33:44.772925 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:44.772968 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:33:44.773037 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:44.773316 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:44.773352 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:44.773356 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:33:44.773508 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 12:33:44.773753 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":134,"threads_started":1}
I20260926 12:33:44.775782 15575 heartbeater.cc:347] Connected to a master server at 127.10.98.62:46253
I20260926 12:33:44.776032 15575 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:44.776526 15575 heartbeater.cc:511] Master 127.10.98.62:46253 requested a full tablet report, sending...
I20260926 12:33:44.777594 15256 ts_manager.cc:194] Registered new tserver with Master: 4426404c20d8454fb3b25198453f6431 (127.10.98.2:35867)
W20260926 12:33:44.883173 15578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:44.883452 15578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:44.883507 15578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:44.887653 15578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:44.887719 15578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:44.887809 15578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:33:44.892809 15578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:37489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:46253
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:44.893905 15578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:44.895241 15578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:33:44.904776 15578 server_base.cc:1034] running on GCE node
W20260926 12:33:44.904728 15584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:44.904785 15586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:44.904973 15583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:44.905397 15578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:44.906062 15578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:44.907255 15578 hybrid_clock.cc:648] HybridClock initialized: now 1790426024907232 us; error 38 us; skew 500 ppm
I20260926 12:33:44.909070 15578 webserver.cc:460] Webserver started at http://127.10.98.3:32867/ using document root <none> and password file <none>
I20260926 12:33:44.909670 15578 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:44.909739 15578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:44.909946 15578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:44.912006 15578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "fa14fea8a83c4d4eb0891eded7d4190e"
format_stamp: "Formatted at 2026-09-26 12:33:44 on dist-test-slave-t7tk"
server_key: "4a8dce96d447a9cd5df8bc440599b288"
server_key_iv: "edbb885c85a388165d6c90023a32244e"
server_key_version: "encryptionkey@0"
I20260926 12:33:44.912531 15578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "fa14fea8a83c4d4eb0891eded7d4190e"
format_stamp: "Formatted at 2026-09-26 12:33:44 on dist-test-slave-t7tk"
server_key: "4a8dce96d447a9cd5df8bc440599b288"
server_key_iv: "edbb885c85a388165d6c90023a32244e"
server_key_version: "encryptionkey@0"
I20260926 12:33:44.916128 15578 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 12:33:44.918713 15592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:44.920017 15578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:33:44.920140 15578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "fa14fea8a83c4d4eb0891eded7d4190e"
format_stamp: "Formatted at 2026-09-26 12:33:44 on dist-test-slave-t7tk"
server_key: "4a8dce96d447a9cd5df8bc440599b288"
server_key_iv: "edbb885c85a388165d6c90023a32244e"
server_key_version: "encryptionkey@0"
I20260926 12:33:44.920275 15578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:44.924901 15578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:44.925570 15578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:44.925722 15578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:44.926244 15578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:44.927122 15578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:44.927166 15578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:44.927217 15578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:44.927249 15578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:44.935766 15578 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:33667
I20260926 12:33:44.935928 15705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:33667 every 8 connection(s)
I20260926 12:33:44.936686 15578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:33:44.937909 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 15578
I20260926 12:33:44.938023 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:33:44.938310 10632 external_mini_cluster.cc:1428] Setting key 60a7e4bcfe6d83e777d2966e2fb398a2
I20260926 12:33:44.943254 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:44.938742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:50963 (local address 127.10.98.62:46253)
0926 12:33:44.938877 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:44.938881 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:44.940282 (+ 1401us) server_negotiation.cc:407] Connection header received
0926 12:33:44.941644 (+ 1362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:44.941647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:44.941683 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:44.941752 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:44.942773 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:44.942776 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:44.942778 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:33:44.942789 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:44.942811 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:33:44.942819 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:44.942904 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:44.943105 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 12:33:44.943147 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 12:33:44.944110 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:44.939341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:46253 (local address 127.10.98.3:50963)
0926 12:33:44.940117 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:44.940172 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:44.941454 (+ 1282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:44.941888 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:44.941894 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:44.941986 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:44.942513 (+ 527us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:33:44.942514 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:44.942554 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:33:44.942628 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:44.942934 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:44.942943 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:44.942948 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 12:33:44.943104 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 12:33:44.943300 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":125,"threads_started":1}
I20260926 12:33:44.945212 15706 heartbeater.cc:347] Connected to a master server at 127.10.98.62:46253
I20260926 12:33:44.945436 15706 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:44.945829 15706 heartbeater.cc:511] Master 127.10.98.62:46253 requested a full tablet report, sending...
I20260926 12:33:44.946841 15256 ts_manager.cc:194] Registered new tserver with Master: fa14fea8a83c4d4eb0891eded7d4190e (127.10.98.3:33667)
I20260926 12:33:44.952294 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:33:44.955559 15295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:44.954245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50194 (local address 127.10.98.62:46253)
0926 12:33:44.954368 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:44.954371 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:44.954535 (+ 164us) server_negotiation.cc:407] Connection header received
0926 12:33:44.954649 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:44.954652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:44.954686 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:44.954751 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:44.955290 (+ 539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:44.955293 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:44.955295 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:33:44.955305 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:44.955325 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:33:44.955332 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:44.955408 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:44.955429 (+ 21us) server_negotiation.cc:299] Negotiation successful
0926 12:33:44.955461 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:33:44.958488 15256 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:33:44.961664 15256 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:33:44.973119 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:44.970836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:34997 (local address 127.0.0.1:58918)
0926 12:33:44.971125 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:44.971137 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:44.971207 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:44.971795 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:44.971798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:44.971812 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:44.972237 (+ 425us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:33:44.972239 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:44.972262 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:33:44.972340 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:44.972757 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:44.972766 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:44.972773 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 12:33:44.972891 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 12:33:44.972961 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":102,"threads_started":1}
I20260926 12:33:44.973165 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:44.970998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58918 (local address 127.10.98.1:34997)
0926 12:33:44.971436 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:44.971445 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:44.971463 (+ 18us) server_negotiation.cc:407] Connection header received
0926 12:33:44.971507 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:44.971511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:44.971648 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:44.972022 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:44.972575 (+ 553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:44.972581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:44.972593 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:33:44.972602 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:44.972639 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:33:44.972663 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:44.972774 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:44.972916 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 12:33:44.972991 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":132,"threads_started":1}
I20260926 12:33:44.973315 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:44.970347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:33667 (local address 127.0.0.1:57186)
0926 12:33:44.970589 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:44.970602 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:44.970682 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:44.972306 (+ 1624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:44.972309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:44.972325 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:44.972684 (+ 359us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:33:44.972686 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:44.972696 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:33:44.972721 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:44.973072 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:44.973075 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:44.973078 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 12:33:44.973167 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 12:33:44.973200 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":92,"threads_started":1}
I20260926 12:33:44.973505 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:44.970056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:35867 (local address 127.0.0.1:45246)
0926 12:33:44.970471 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:44.970517 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:44.970613 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:44.971589 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:44.971595 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:44.971672 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:44.972304 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:33:44.972320 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:44.972328 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:33:44.972347 (+ 19us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:44.973306 (+ 959us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:44.973309 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:44.973313 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:33:44.973364 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 12:33:44.973399 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":73,"threads_started":1}
I20260926 12:33:44.973875 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:44.970846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57186 (local address 127.10.98.3:33667)
0926 12:33:44.971233 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:44.971245 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:44.971279 (+ 34us) server_negotiation.cc:407] Connection header received
0926 12:33:44.971331 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:44.971336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:44.971523 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:44.971648 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:44.972840 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:44.972846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:44.972858 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:33:44.972869 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:44.972911 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:33:44.972943 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:44.973032 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:44.973670 (+ 638us) server_negotiation.cc:299] Negotiation successful
0926 12:33:44.973742 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":109,"threads_started":1}
I20260926 12:33:44.973875 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:44.970156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45246 (local address 127.10.98.2:35867)
0926 12:33:44.971128 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:44.971140 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:44.971166 (+ 26us) server_negotiation.cc:407] Connection header received
0926 12:33:44.971219 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:44.971224 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:44.971430 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:44.972495 (+ 1065us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:44.973087 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:44.973092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:44.973127 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:33:44.973138 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:44.973173 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:33:44.973199 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:44.973658 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:44.973703 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 12:33:44.973768 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":860,"thread_start_us":101,"threads_started":1}
I20260926 12:33:44.976257 15509 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9cc1b7351c4a159a7e63b64b6b791f (DEFAULT_TABLE table=test-table [id=871cb93b99d048bf9f04c450cdd153ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:44.976363 15640 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9cc1b7351c4a159a7e63b64b6b791f (DEFAULT_TABLE table=test-table [id=871cb93b99d048bf9f04c450cdd153ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:44.976363 15377 tablet_service.cc:1467] Processing CreateTablet for tablet 8a9cc1b7351c4a159a7e63b64b6b791f (DEFAULT_TABLE table=test-table [id=871cb93b99d048bf9f04c450cdd153ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:44.977232 15509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9cc1b7351c4a159a7e63b64b6b791f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:44.977360 15377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9cc1b7351c4a159a7e63b64b6b791f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:44.977360 15640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a9cc1b7351c4a159a7e63b64b6b791f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:44.983338 15723 tablet_bootstrap.cc:492] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431: Bootstrap starting.
I20260926 12:33:44.984304 15724 tablet_bootstrap.cc:492] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e: Bootstrap starting.
I20260926 12:33:44.985189 15725 tablet_bootstrap.cc:492] T 8a9cc1b7351c4a159a7e63b64b6b791f P 6a98514e15b3441a884293f7094353d3: Bootstrap starting.
I20260926 12:33:44.986002 15723 tablet_bootstrap.cc:654] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:44.986395 15724 tablet_bootstrap.cc:654] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:44.987043 15725 tablet_bootstrap.cc:654] T 8a9cc1b7351c4a159a7e63b64b6b791f P 6a98514e15b3441a884293f7094353d3: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:44.987032 15723 log.cc:824] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:44.987157 15724 log.cc:824] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:44.987890 15725 log.cc:824] T 8a9cc1b7351c4a159a7e63b64b6b791f P 6a98514e15b3441a884293f7094353d3: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:44.989125 15724 tablet_bootstrap.cc:492] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e: No bootstrap required, opened a new log
I20260926 12:33:44.989125 15723 tablet_bootstrap.cc:492] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431: No bootstrap required, opened a new log
I20260926 12:33:44.989317 15723 ts_tablet_manager.cc:1397] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:33:44.989317 15724 ts_tablet_manager.cc:1397] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260926 12:33:44.989557 15725 tablet_bootstrap.cc:492] T 8a9cc1b7351c4a159a7e63b64b6b791f P 6a98514e15b3441a884293f7094353d3: No bootstrap required, opened a new log
I20260926 12:33:44.989727 15725 ts_tablet_manager.cc:1397] T 8a9cc1b7351c4a159a7e63b64b6b791f P 6a98514e15b3441a884293f7094353d3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:33:44.993474 15725 raft_consensus.cc:348] T 8a9cc1b7351c4a159a7e63b64b6b791f P 6a98514e15b3441a884293f7094353d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4426404c20d8454fb3b25198453f6431" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35867 } } peers { permanent_uuid: "fa14fea8a83c4d4eb0891eded7d4190e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33667 } } peers { permanent_uuid: "6a98514e15b3441a884293f7094353d3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34997 } }
I20260926 12:33:44.993700 15725 raft_consensus.cc:374] T 8a9cc1b7351c4a159a7e63b64b6b791f P 6a98514e15b3441a884293f7094353d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:44.993750 15725 raft_consensus.cc:729] T 8a9cc1b7351c4a159a7e63b64b6b791f P 6a98514e15b3441a884293f7094353d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a98514e15b3441a884293f7094353d3, State: Initialized, Role: FOLLOWER
I20260926 12:33:44.993696 15724 raft_consensus.cc:348] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4426404c20d8454fb3b25198453f6431" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35867 } } peers { permanent_uuid: "fa14fea8a83c4d4eb0891eded7d4190e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33667 } } peers { permanent_uuid: "6a98514e15b3441a884293f7094353d3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34997 } }
I20260926 12:33:44.993815 15723 raft_consensus.cc:348] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4426404c20d8454fb3b25198453f6431" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35867 } } peers { permanent_uuid: "fa14fea8a83c4d4eb0891eded7d4190e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33667 } } peers { permanent_uuid: "6a98514e15b3441a884293f7094353d3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34997 } }
I20260926 12:33:44.993978 15724 raft_consensus.cc:374] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:44.994011 15723 raft_consensus.cc:374] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:44.994026 15724 raft_consensus.cc:729] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa14fea8a83c4d4eb0891eded7d4190e, State: Initialized, Role: FOLLOWER
I20260926 12:33:44.994059 15723 raft_consensus.cc:729] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4426404c20d8454fb3b25198453f6431, State: Initialized, Role: FOLLOWER
I20260926 12:33:44.994366 15725 consensus_queue.cc:260] T 8a9cc1b7351c4a159a7e63b64b6b791f P 6a98514e15b3441a884293f7094353d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4426404c20d8454fb3b25198453f6431" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35867 } } peers { permanent_uuid: "fa14fea8a83c4d4eb0891eded7d4190e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33667 } } peers { permanent_uuid: "6a98514e15b3441a884293f7094353d3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34997 } }
I20260926 12:33:44.994446 15723 consensus_queue.cc:260] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4426404c20d8454fb3b25198453f6431" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35867 } } peers { permanent_uuid: "fa14fea8a83c4d4eb0891eded7d4190e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33667 } } peers { permanent_uuid: "6a98514e15b3441a884293f7094353d3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34997 } }
I20260926 12:33:44.994474 15724 consensus_queue.cc:260] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4426404c20d8454fb3b25198453f6431" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35867 } } peers { permanent_uuid: "fa14fea8a83c4d4eb0891eded7d4190e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33667 } } peers { permanent_uuid: "6a98514e15b3441a884293f7094353d3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34997 } }
I20260926 12:33:44.995018 15444 heartbeater.cc:503] Master 127.10.98.62:46253 was elected leader, sending a full tablet report...
I20260926 12:33:44.995379 15725 ts_tablet_manager.cc:1428] T 8a9cc1b7351c4a159a7e63b64b6b791f P 6a98514e15b3441a884293f7094353d3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:33:44.995400 15724 ts_tablet_manager.cc:1428] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:33:44.995711 15706 heartbeater.cc:503] Master 127.10.98.62:46253 was elected leader, sending a full tablet report...
I20260926 12:33:44.996137 15723 ts_tablet_manager.cc:1428] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 12:33:44.996452 15575 heartbeater.cc:503] Master 127.10.98.62:46253 was elected leader, sending a full tablet report...
W20260926 12:33:45.017995 15576 tablet.cc:2369] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:33:45.059077 15730 raft_consensus.cc:482] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:33:45.059305 15730 raft_consensus.cc:504] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4426404c20d8454fb3b25198453f6431" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35867 } } peers { permanent_uuid: "fa14fea8a83c4d4eb0891eded7d4190e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33667 } } peers { permanent_uuid: "6a98514e15b3441a884293f7094353d3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34997 } }
I20260926 12:33:45.060626 15730 leader_election.cc:288] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4426404c20d8454fb3b25198453f6431 (127.10.98.2:35867), 6a98514e15b3441a884293f7094353d3 (127.10.98.1:34997)
I20260926 12:33:45.062646 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:45.061055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:40245 (local address 127.10.98.2:35867)
0926 12:33:45.061205 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:45.061210 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:45.061276 (+ 66us) server_negotiation.cc:407] Connection header received
0926 12:33:45.061465 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:45.061467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:45.061507 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:45.061583 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:45.062291 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:45.062294 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:45.062297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:33:45.062309 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:45.062328 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:33:45.062337 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:45.062415 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:45.062484 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 12:33:45.062523 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 12:33:45.062646 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:45.060937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:35867 (local address 127.10.98.3:40245)
0926 12:33:45.061198 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:45.061212 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:45.061323 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:45.061625 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:45.061628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:45.061644 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:45.062118 (+ 474us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:33:45.062120 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:45.062130 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:33:45.062155 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:45.062409 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:45.062412 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:45.062415 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 12:33:45.062486 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 12:33:45.062520 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260926 12:33:45.062750 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:45.061068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:57667 (local address 127.10.98.1:34997)
0926 12:33:45.061211 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:45.061216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:45.061493 (+ 277us) server_negotiation.cc:407] Connection header received
0926 12:33:45.061574 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:45.061577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:45.061611 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:45.061672 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:45.062411 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:45.062414 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:45.062416 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:33:45.062427 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:45.062444 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:33:45.062451 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:45.062522 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:45.062592 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 12:33:45.062628 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 12:33:45.062980 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:45.060958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:34997 (local address 127.10.98.3:57667)
0926 12:33:45.061401 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:45.061413 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:45.061494 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:45.061708 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:45.061711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:45.061726 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:45.062241 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:33:45.062243 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:45.062252 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:33:45.062274 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:45.062520 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:45.062522 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:45.062526 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:33:45.062594 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 12:33:45.062627 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":3,"thread_start_us":146,"threads_started":1}
I20260926 12:33:45.063457 15529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9cc1b7351c4a159a7e63b64b6b791f" candidate_uuid: "fa14fea8a83c4d4eb0891eded7d4190e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4426404c20d8454fb3b25198453f6431" is_pre_election: true
I20260926 12:33:45.063537 15398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9cc1b7351c4a159a7e63b64b6b791f" candidate_uuid: "fa14fea8a83c4d4eb0891eded7d4190e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a98514e15b3441a884293f7094353d3" is_pre_election: true
I20260926 12:33:45.063795 15398 raft_consensus.cc:2454] T 8a9cc1b7351c4a159a7e63b64b6b791f P 6a98514e15b3441a884293f7094353d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa14fea8a83c4d4eb0891eded7d4190e in term 0.
I20260926 12:33:45.063795 15529 raft_consensus.cc:2454] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa14fea8a83c4d4eb0891eded7d4190e in term 0.
I20260926 12:33:45.064282 15596 leader_election.cc:302] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a98514e15b3441a884293f7094353d3, fa14fea8a83c4d4eb0891eded7d4190e; no voters:
I20260926 12:33:45.064556 15730 raft_consensus.cc:2781] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:33:45.064639 15730 raft_consensus.cc:482] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:33:45.064685 15730 raft_consensus.cc:3037] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:45.065779 15730 raft_consensus.cc:504] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4426404c20d8454fb3b25198453f6431" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35867 } } peers { permanent_uuid: "fa14fea8a83c4d4eb0891eded7d4190e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33667 } } peers { permanent_uuid: "6a98514e15b3441a884293f7094353d3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34997 } }
I20260926 12:33:45.066115 15730 leader_election.cc:288] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e [CANDIDATE]: Term 1 election: Requested vote from peers 4426404c20d8454fb3b25198453f6431 (127.10.98.2:35867), 6a98514e15b3441a884293f7094353d3 (127.10.98.1:34997)
I20260926 12:33:45.066468 15529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9cc1b7351c4a159a7e63b64b6b791f" candidate_uuid: "fa14fea8a83c4d4eb0891eded7d4190e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4426404c20d8454fb3b25198453f6431"
I20260926 12:33:45.066475 15398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8a9cc1b7351c4a159a7e63b64b6b791f" candidate_uuid: "fa14fea8a83c4d4eb0891eded7d4190e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a98514e15b3441a884293f7094353d3"
I20260926 12:33:45.066597 15529 raft_consensus.cc:3037] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:45.066597 15398 raft_consensus.cc:3037] T 8a9cc1b7351c4a159a7e63b64b6b791f P 6a98514e15b3441a884293f7094353d3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:45.067840 15529 raft_consensus.cc:2454] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa14fea8a83c4d4eb0891eded7d4190e in term 1.
I20260926 12:33:45.067839 15398 raft_consensus.cc:2454] T 8a9cc1b7351c4a159a7e63b64b6b791f P 6a98514e15b3441a884293f7094353d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa14fea8a83c4d4eb0891eded7d4190e in term 1.
I20260926 12:33:45.068167 15596 leader_election.cc:302] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4426404c20d8454fb3b25198453f6431, 6a98514e15b3441a884293f7094353d3, fa14fea8a83c4d4eb0891eded7d4190e; no voters:
I20260926 12:33:45.068326 15730 raft_consensus.cc:2781] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:45.068539 15730 raft_consensus.cc:686] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e [term 1 LEADER]: Becoming Leader. State: Replica: fa14fea8a83c4d4eb0891eded7d4190e, State: Running, Role: LEADER
I20260926 12:33:45.068892 15730 consensus_queue.cc:237] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4426404c20d8454fb3b25198453f6431" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35867 } } peers { permanent_uuid: "fa14fea8a83c4d4eb0891eded7d4190e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33667 } } peers { permanent_uuid: "6a98514e15b3441a884293f7094353d3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34997 } }
I20260926 12:33:45.072023 15254 catalog_manager.cc:5495] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e reported cstate change: term changed from 0 to 1, leader changed from <none> to fa14fea8a83c4d4eb0891eded7d4190e (127.10.98.3). New cstate: current_term: 1 leader_uuid: "fa14fea8a83c4d4eb0891eded7d4190e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4426404c20d8454fb3b25198453f6431" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa14fea8a83c4d4eb0891eded7d4190e" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33667 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6a98514e15b3441a884293f7094353d3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34997 } health_report { overall_health: UNKNOWN } } }
I20260926 12:33:45.097517 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:45.096081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57190 (local address 127.10.98.3:33667)
0926 12:33:45.096221 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:45.096226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:45.096239 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:33:45.096322 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:45.096324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:45.096384 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:45.096461 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:45.097090 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:45.097117 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:45.097120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:33:45.097132 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:45.097152 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:33:45.097161 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:45.097238 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:45.097318 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 12:33:45.097356 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 12:33:45.098079 15445 tablet.cc:2369] T 8a9cc1b7351c4a159a7e63b64b6b791f P 6a98514e15b3441a884293f7094353d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:33:45.104028 15398 raft_consensus.cc:1261] T 8a9cc1b7351c4a159a7e63b64b6b791f P 6a98514e15b3441a884293f7094353d3 [term 1 FOLLOWER]: Refusing update from remote peer fa14fea8a83c4d4eb0891eded7d4190e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:45.104028 15529 raft_consensus.cc:1261] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431 [term 1 FOLLOWER]: Refusing update from remote peer fa14fea8a83c4d4eb0891eded7d4190e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:45.104856 15733 consensus_queue.cc:1035] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a98514e15b3441a884293f7094353d3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:45.105038 15730 consensus_queue.cc:1035] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4426404c20d8454fb3b25198453f6431" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:45.114571 15737 mvcc.cc:205] Tried to move back new op lower bound from 7333584998816419840 to 7333584998685007872. Current Snapshot: MvccSnapshot[applied={T|T < 7333584998816419840}]
I20260926 12:33:45.121547 15256 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:45.121762 15256 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:45.124529 15256 catalog_manager.cc:5782] T 00000000000000000000000000000000 P adc60cff0e6444a2ae38d9724e8bbbff: Sending DeleteTablet for 3 replicas of tablet 8a9cc1b7351c4a159a7e63b64b6b791f
I20260926 12:33:45.125411 15640 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9cc1b7351c4a159a7e63b64b6b791f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:45 UTC) from {username='slave'} at 127.0.0.1:57186
I20260926 12:33:45.125433 15377 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9cc1b7351c4a159a7e63b64b6b791f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:45 UTC) from {username='slave'} at 127.0.0.1:58918
I20260926 12:33:45.125910 15509 tablet_service.cc:1514] Processing DeleteTablet for tablet 8a9cc1b7351c4a159a7e63b64b6b791f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:45 UTC) from {username='slave'} at 127.0.0.1:45246
I20260926 12:33:45.125994 15746 tablet_replica.cc:330] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e: stopping tablet replica
I20260926 12:33:45.126247 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 15316
I20260926 12:33:45.126331 15746 raft_consensus.cc:2229] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:33:45.126844 15746 raft_consensus.cc:2258] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:45.127993 15748 tablet_replica.cc:330] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431: stopping tablet replica
I20260926 12:33:45.128322 15748 raft_consensus.cc:2229] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:45.128656 15748 raft_consensus.cc:2258] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:45.129072 15746 ts_tablet_manager.cc:1905] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:45.130232 15748 ts_tablet_manager.cc:1905] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:45.132118 15746 ts_tablet_manager.cc:1918] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:33:45.132232 15746 log.cc:1196] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/8a9cc1b7351c4a159a7e63b64b6b791f
I20260926 12:33:45.132570 15746 ts_tablet_manager.cc:1939] T 8a9cc1b7351c4a159a7e63b64b6b791f P fa14fea8a83c4d4eb0891eded7d4190e: Deleting consensus metadata
I20260926 12:33:45.132855 15748 ts_tablet_manager.cc:1918] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:33:45.132952 15748 log.cc:1196] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/8a9cc1b7351c4a159a7e63b64b6b791f
I20260926 12:33:45.133260 15748 ts_tablet_manager.cc:1939] T 8a9cc1b7351c4a159a7e63b64b6b791f P 4426404c20d8454fb3b25198453f6431: Deleting consensus metadata
I20260926 12:33:45.133949 15242 catalog_manager.cc:4841] TS fa14fea8a83c4d4eb0891eded7d4190e (127.10.98.3:33667): tablet 8a9cc1b7351c4a159a7e63b64b6b791f (table test-table [id=871cb93b99d048bf9f04c450cdd153ef]) successfully deleted
I20260926 12:33:45.134454 15241 catalog_manager.cc:4841] TS 4426404c20d8454fb3b25198453f6431 (127.10.98.2:35867): tablet 8a9cc1b7351c4a159a7e63b64b6b791f (table test-table [id=871cb93b99d048bf9f04c450cdd153ef]) successfully deleted
W20260926 12:33:45.134976 15243 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.98.1:34997 (error 108)
I20260926 12:33:45.135193 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 15447
I20260926 12:33:45.137131 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:45.136775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:34997 (local address 127.0.0.1:58932)
0926 12:33:45.136921 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:45.137005 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.1:34997: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 12:33:45.137295 15243 catalog_manager.cc:4584] TS 6a98514e15b3441a884293f7094353d3 (127.10.98.1:34997): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8a9cc1b7351c4a159a7e63b64b6b791f: Network error: Client connection negotiation failed: client connection to 127.10.98.1:34997: connect: Connection refused (error 111)
I20260926 12:33:45.140837 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 15578
I20260926 12:33:45.145900 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 15225
2026-09-26T12:33:45Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (970 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 12:33:45.166437 10632 mini_oidc.cc:152] Starting JWKS server
I20260926 12:33:45.168018 10632 webserver.cc:460] Webserver started at https://127.0.0.1:39559/ using document root <none> and password file <none>
I20260926 12:33:45.168172 10632 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 12:33:45.168427 10632 webserver.cc:460] Webserver started at http://127.0.0.1:41131/ using document root <none> and password file <none>
2026-09-26T12:33:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:33:45Z Disabled control of system clock
I20260926 12:33:45.176666 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:38557
--webserver_interface=127.10.98.62
--webserver_port=0
--builtin_ntp_servers=127.10.98.20:33885
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:38557
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39559/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425997379116-10632-0/testchainca.cert with env {}
W20260926 12:33:45.286980 15755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:45.287251 15755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:45.287307 15755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:45.291473 15755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:45.291538 15755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:45.291559 15755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:45.291580 15755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:45.291616 15755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 12:33:45.291646 15755 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 12:33:45.291666 15755 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39559/jwks.json
I20260926 12:33:45.297021 15755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:33885
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:38557
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:38557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39559/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425997379116-10632-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:45.298048 15755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:45.299002 15755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:45.307334 15760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:45.307343 15761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:45.307459 15763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:45.307570 15755 server_base.cc:1034] running on GCE node
I20260926 12:33:45.308398 15755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:45.309508 15755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:45.310681 15755 hybrid_clock.cc:648] HybridClock initialized: now 1790426025310656 us; error 39 us; skew 500 ppm
I20260926 12:33:45.320998 15755 webserver.cc:460] Webserver started at http://127.10.98.62:39137/ using document root <none> and password file <none>
I20260926 12:33:45.321645 15755 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:45.321730 15755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:45.321942 15755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:45.324067 15755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "7c7907ae7b584b3d80ac77cdcdceb706"
format_stamp: "Formatted at 2026-09-26 12:33:45 on dist-test-slave-t7tk"
server_key: "216d5e6cd451f5e3a161e332f2a9ea2c"
server_key_iv: "cc50c3e2db121475ac61ca55753978eb"
server_key_version: "encryptionkey@0"
I20260926 12:33:45.324669 15755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "7c7907ae7b584b3d80ac77cdcdceb706"
format_stamp: "Formatted at 2026-09-26 12:33:45 on dist-test-slave-t7tk"
server_key: "216d5e6cd451f5e3a161e332f2a9ea2c"
server_key_iv: "cc50c3e2db121475ac61ca55753978eb"
server_key_version: "encryptionkey@0"
I20260926 12:33:45.328616 15755 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:33:45.331329 15769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:45.332643 15755 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:33:45.332765 15755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "7c7907ae7b584b3d80ac77cdcdceb706"
format_stamp: "Formatted at 2026-09-26 12:33:45 on dist-test-slave-t7tk"
server_key: "216d5e6cd451f5e3a161e332f2a9ea2c"
server_key_iv: "cc50c3e2db121475ac61ca55753978eb"
server_key_version: "encryptionkey@0"
I20260926 12:33:45.332872 15755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:45.349169 15755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:45.349923 15755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:45.350113 15755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:45.356766 15755 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:38557
I20260926 12:33:45.356784 15821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:38557 every 8 connection(s)
I20260926 12:33:45.357712 15755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:33:45.360803 15822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:45.362138 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 15755
I20260926 12:33:45.362287 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:33:45.362578 10632 external_mini_cluster.cc:1428] Setting key 0b477446fe7bdfc98b4bc918d883c006
I20260926 12:33:45.367112 15822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706: Bootstrap starting.
I20260926 12:33:45.369366 15822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:45.370240 15822 log.cc:824] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:45.372013 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:45.363932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55248 (local address 127.10.98.62:38557)
0926 12:33:45.364590 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:45.364607 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:45.364644 (+ 37us) server_negotiation.cc:407] Connection header received
0926 12:33:45.365517 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:45.365538 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:45.365994 (+ 456us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:45.366327 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:45.366796 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:45.367833 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:45.368764 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:45.369275 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:45.369418 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:45.370214 (+ 796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:45.370219 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:45.370232 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:33:45.370690 (+ 458us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:45.370764 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:33:45.370791 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:45.370901 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:45.371031 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 12:33:45.371297 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":133,"threads_started":1}
I20260926 12:33:45.372196 15822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706: No bootstrap required, opened a new log
I20260926 12:33:45.374645 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:33:45.375808 15822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7907ae7b584b3d80ac77cdcdceb706" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38557 } }
I20260926 12:33:45.376022 15822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:45.376070 15822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c7907ae7b584b3d80ac77cdcdceb706, State: Initialized, Role: FOLLOWER
I20260926 12:33:45.376447 15822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7907ae7b584b3d80ac77cdcdceb706" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38557 } }
I20260926 12:33:45.376565 15822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:33:45.376624 15822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:33:45.376717 15822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:45.377586 15822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7907ae7b584b3d80ac77cdcdceb706" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38557 } }
I20260926 12:33:45.377883 15822 leader_election.cc:302] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c7907ae7b584b3d80ac77cdcdceb706; no voters:
I20260926 12:33:45.378157 15822 leader_election.cc:288] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:33:45.378280 15827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:45.378504 15827 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706 [term 1 LEADER]: Becoming Leader. State: Replica: 7c7907ae7b584b3d80ac77cdcdceb706, State: Running, Role: LEADER
I20260926 12:33:45.378829 15827 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7907ae7b584b3d80ac77cdcdceb706" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38557 } }
I20260926 12:33:45.379259 15822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:45.380277 15828 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c7907ae7b584b3d80ac77cdcdceb706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7907ae7b584b3d80ac77cdcdceb706" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38557 } } }
I20260926 12:33:45.380403 15828 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:45.380689 15829 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c7907ae7b584b3d80ac77cdcdceb706. Latest consensus state: current_term: 1 leader_uuid: "7c7907ae7b584b3d80ac77cdcdceb706" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c7907ae7b584b3d80ac77cdcdceb706" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38557 } } }
I20260926 12:33:45.380760 15836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:33:45.382054 15829 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:45.383433 15836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:33:45.388590 15836 catalog_manager.cc:1347] Generated new cluster ID: 97639acfd9c74a0bab3dfbeb869c2ac0
I20260926 12:33:45.388653 15836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:33:45.430058 15836 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:33:45.431344 15836 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:33:45.442310 15836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c7907ae7b584b3d80ac77cdcdceb706: Generated new TSK 0
I20260926 12:33:45.443260 15836 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 12:33:45.480376 15852 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 12:33:45.480515 15852 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 12:33:45.480969 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:45.463702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55258 (local address 127.10.98.62:38557)
0926 12:33:45.463884 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:45.463889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:45.464199 (+ 310us) server_negotiation.cc:407] Connection header received
0926 12:33:45.464452 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:45.464455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:45.464510 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:45.464604 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0926 12:33:45.464931 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:45.465516 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:45.466713 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:45.467040 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:45.467142 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:45.467548 (+ 406us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 12:33:45.480562 (+ 13014us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 12:33:45.480843 (+ 281us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55258: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":94}
W20260926 12:33:45.481050 15825 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55258: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 12:33:45.482635 15851 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:45.463573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38557 (local address 127.0.0.1:55258)
0926 12:33:45.464086 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:45.464101 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:45.464247 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:45.464643 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:45.464647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:45.464655 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 12:33:45.464797 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:45.464803 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:45.465628 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:45.465631 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:45.466577 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:45.466587 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:45.467173 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:45.467178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:45.467276 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:45.467337 (+ 61us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 12:33:45.480819 (+ 13482us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 12:33:45.481088 (+ 269us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.98.62:38557: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":204,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.10.98.62:38557: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39559/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 12:33:45.483385 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 15755
2026-09-26T12:33:45Z chronyd exiting
I20260926 12:33:45.520223 10632 test_util.cc:175] -----------------------------------------------
I20260926 12:33:45.520310 10632 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790425997379116-10632-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (355 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 12:33:45.521427 10632 mini_oidc.cc:152] Starting JWKS server
I20260926 12:33:45.522253 10632 webserver.cc:460] Webserver started at https://127.0.0.1:40539/ using document root <none> and password file <none>
I20260926 12:33:45.522339 10632 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 12:33:45.522590 10632 webserver.cc:460] Webserver started at http://127.0.0.1:34719/ using document root <none> and password file <none>
2026-09-26T12:33:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:33:45Z Disabled control of system clock
I20260926 12:33:45.531198 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:40873
--webserver_interface=127.10.98.62
--webserver_port=0
--builtin_ntp_servers=127.10.98.20:46231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:40873
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40539/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425997379116-10632-0/testchainca.cert with env {}
W20260926 12:33:45.646057 15858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:45.646371 15858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:45.646415 15858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:45.651069 15858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:45.651165 15858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:45.651180 15858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:45.651196 15858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:45.651209 15858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 12:33:45.651225 15858 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 12:33:45.651237 15858 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40539/jwks.json
I20260926 12:33:45.657027 15858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:46231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:40873
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:40873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40539/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425997379116-10632-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15858
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:45.658191 15858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:45.659224 15858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:45.668391 15863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:45.668394 15864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:45.668490 15858 server_base.cc:1034] running on GCE node
W20260926 12:33:45.668429 15866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:45.669261 15858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:45.670478 15858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:45.671672 15858 hybrid_clock.cc:648] HybridClock initialized: now 1790426025671643 us; error 42 us; skew 500 ppm
I20260926 12:33:45.678857 15858 webserver.cc:460] Webserver started at http://127.10.98.62:42123/ using document root <none> and password file <none>
I20260926 12:33:45.679513 15858 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:45.679590 15858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:45.679782 15858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:45.682019 15858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "d843f08773a64327ae49e828e907e868"
format_stamp: "Formatted at 2026-09-26 12:33:45 on dist-test-slave-t7tk"
server_key: "e8cf240182f29d6ac4e2d5312554d360"
server_key_iv: "365fde73572b1b163d14a71ea58a27b9"
server_key_version: "encryptionkey@0"
I20260926 12:33:45.682570 15858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "d843f08773a64327ae49e828e907e868"
format_stamp: "Formatted at 2026-09-26 12:33:45 on dist-test-slave-t7tk"
server_key: "e8cf240182f29d6ac4e2d5312554d360"
server_key_iv: "365fde73572b1b163d14a71ea58a27b9"
server_key_version: "encryptionkey@0"
I20260926 12:33:45.686371 15858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:33:45.689052 15872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:45.690572 15858 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:33:45.690726 15858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "d843f08773a64327ae49e828e907e868"
format_stamp: "Formatted at 2026-09-26 12:33:45 on dist-test-slave-t7tk"
server_key: "e8cf240182f29d6ac4e2d5312554d360"
server_key_iv: "365fde73572b1b163d14a71ea58a27b9"
server_key_version: "encryptionkey@0"
I20260926 12:33:45.690853 15858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:45.709672 15858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:45.710409 15858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:45.710611 15858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:45.717430 15858 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:40873
I20260926 12:33:45.717456 15924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:40873 every 8 connection(s)
I20260926 12:33:45.718366 15858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:33:45.721571 15925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:45.727063 15925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868: Bootstrap starting.
I20260926 12:33:45.727270 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 15858
I20260926 12:33:45.727391 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:33:45.727692 10632 external_mini_cluster.cc:1428] Setting key c2e50e2ba8d8b740eec8ff1b0f7ef94a
I20260926 12:33:45.729871 15925 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:45.730585 15925 log.cc:824] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:45.732384 15925 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868: No bootstrap required, opened a new log
I20260926 12:33:45.735952 15925 raft_consensus.cc:348] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843f08773a64327ae49e828e907e868" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40873 } }
I20260926 12:33:45.736240 15925 raft_consensus.cc:374] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:45.736299 15925 raft_consensus.cc:729] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d843f08773a64327ae49e828e907e868, State: Initialized, Role: FOLLOWER
I20260926 12:33:45.736733 15925 consensus_queue.cc:260] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843f08773a64327ae49e828e907e868" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40873 } }
I20260926 12:33:45.736863 15925 raft_consensus.cc:388] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:33:45.736936 15925 raft_consensus.cc:482] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:33:45.736996 15928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:45.729047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34852 (local address 127.10.98.62:40873)
0926 12:33:45.729735 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:45.729749 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:45.729785 (+ 36us) server_negotiation.cc:407] Connection header received
0926 12:33:45.730537 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:45.730560 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:45.731018 (+ 458us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:45.731332 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:45.731769 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:45.732768 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:45.733697 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:45.734194 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:45.734339 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:45.735169 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:45.735174 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:45.735187 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:33:45.735596 (+ 409us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:45.735650 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:33:45.735671 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:45.735756 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:45.735895 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 12:33:45.736150 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":136,"threads_started":1}
I20260926 12:33:45.737027 15925 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:45.738137 15925 raft_consensus.cc:504] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843f08773a64327ae49e828e907e868" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40873 } }
I20260926 12:33:45.738451 15925 leader_election.cc:302] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d843f08773a64327ae49e828e907e868; no voters:
I20260926 12:33:45.738737 15925 leader_election.cc:288] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:33:45.738860 15930 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:45.739058 15930 raft_consensus.cc:686] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868 [term 1 LEADER]: Becoming Leader. State: Replica: d843f08773a64327ae49e828e907e868, State: Running, Role: LEADER
I20260926 12:33:45.739352 15930 consensus_queue.cc:237] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843f08773a64327ae49e828e907e868" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40873 } }
I20260926 12:33:45.739948 15925 sys_catalog.cc:564] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:45.740028 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:33:45.740586 15930 sys_catalog.cc:455] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d843f08773a64327ae49e828e907e868. Latest consensus state: current_term: 1 leader_uuid: "d843f08773a64327ae49e828e907e868" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843f08773a64327ae49e828e907e868" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40873 } } }
I20260926 12:33:45.740701 15930 sys_catalog.cc:458] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:45.741063 15931 sys_catalog.cc:455] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d843f08773a64327ae49e828e907e868" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d843f08773a64327ae49e828e907e868" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40873 } } }
I20260926 12:33:45.741938 15931 sys_catalog.cc:458] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868 [sys.catalog]: This master's current role is: LEADER
W20260926 12:33:45.744328 15945 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:33:45.744393 15945 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:33:45.744477 15937 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:33:45.745186 15937 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:33:45.750666 15937 catalog_manager.cc:1347] Generated new cluster ID: ffc150e4ff6847bca32022ff1ef28dfb
I20260926 12:33:45.750761 15937 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:33:45.758309 15937 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:33:45.759590 15937 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:33:45.768723 15937 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d843f08773a64327ae49e828e907e868: Generated new TSK 0
I20260926 12:33:45.769636 15937 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 12:33:45.809618 15954 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:45.782433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40873 (local address 127.0.0.1:34860)
0926 12:33:45.782729 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:45.782744 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:45.782868 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:45.783150 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:45.783153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:45.783161 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0926 12:33:45.783283 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:45.783290 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:45.784848 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:45.784852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:45.785795 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:45.785808 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:45.786596 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:45.786602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:45.786716 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:45.786749 (+ 33us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 12:33:45.809163 (+ 22414us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 12:33:45.809388 (+ 225us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.98.62:40873: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":104,"threads_started":1}
I20260926 12:33:45.809607 15928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:45.782541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34860 (local address 127.10.98.62:40873)
0926 12:33:45.782670 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:45.782675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:45.782821 (+ 146us) server_negotiation.cc:407] Connection header received
0926 12:33:45.782994 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:45.782997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:45.783044 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:45.783135 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0926 12:33:45.783392 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:45.784710 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:45.785995 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:45.786421 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:45.786525 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:45.786953 (+ 428us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 12:33:45.808864 (+ 21911us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 12:33:45.809196 (+ 332us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34860: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":55}
W20260926 12:33:45.809739 15928 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34860: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40539/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
E20260926 12:33:45.810266 15955 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 12:33:45.810336 15955 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 12:33:45.810818 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 15858
2026-09-26T12:33:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (330 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 12:33:45.851759 10632 mini_oidc.cc:152] Starting JWKS server
I20260926 12:33:45.852635 10632 webserver.cc:460] Webserver started at https://127.0.0.1:41571/ using document root <none> and password file <none>
I20260926 12:33:45.852735 10632 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 12:33:45.852991 10632 webserver.cc:460] Webserver started at http://127.0.0.1:41129/ using document root <none> and password file <none>
2026-09-26T12:33:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:33:45Z Disabled control of system clock
I20260926 12:33:45.861173 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:36011
--webserver_interface=127.10.98.62
--webserver_port=0
--builtin_ntp_servers=127.10.98.20:46571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:36011
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41571/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 12:33:45.968670 15961 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:45.968955 15961 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:45.968994 15961 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:45.972985 15961 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:45.973059 15961 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:45.973073 15961 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:45.973088 15961 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:45.973125 15961 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 12:33:45.973146 15961 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 12:33:45.973174 15961 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41571/jwks.json
I20260926 12:33:45.977968 15961 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:46571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:36011
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:36011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41571/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15961
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:45.979032 15961 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:45.980360 15961 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:45.988551 15966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:45.988592 15969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:45.988545 15967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:45.988588 15961 server_base.cc:1034] running on GCE node
I20260926 12:33:45.989348 15961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:45.990450 15961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:45.991626 15961 hybrid_clock.cc:648] HybridClock initialized: now 1790426025991591 us; error 46 us; skew 500 ppm
I20260926 12:33:45.993465 15961 webserver.cc:460] Webserver started at http://127.10.98.62:37025/ using document root <none> and password file <none>
I20260926 12:33:45.994010 15961 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:45.994078 15961 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:45.994299 15961 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:45.996402 15961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "e97e2c28ed1a46cf83689570b5dfc97a"
format_stamp: "Formatted at 2026-09-26 12:33:45 on dist-test-slave-t7tk"
server_key: "9e5cbaf121acce3b880b47765f65ffd6"
server_key_iv: "e7aeb2c20859e1bc0e7edc5d01a068a9"
server_key_version: "encryptionkey@0"
I20260926 12:33:45.996944 15961 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "e97e2c28ed1a46cf83689570b5dfc97a"
format_stamp: "Formatted at 2026-09-26 12:33:45 on dist-test-slave-t7tk"
server_key: "9e5cbaf121acce3b880b47765f65ffd6"
server_key_iv: "e7aeb2c20859e1bc0e7edc5d01a068a9"
server_key_version: "encryptionkey@0"
I20260926 12:33:46.000578 15961 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:33:46.002945 15975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:46.004168 15961 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:33:46.004279 15961 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "e97e2c28ed1a46cf83689570b5dfc97a"
format_stamp: "Formatted at 2026-09-26 12:33:45 on dist-test-slave-t7tk"
server_key: "9e5cbaf121acce3b880b47765f65ffd6"
server_key_iv: "e7aeb2c20859e1bc0e7edc5d01a068a9"
server_key_version: "encryptionkey@0"
I20260926 12:33:46.004374 15961 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:46.021577 15961 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:46.022315 15961 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:46.022533 15961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:46.029177 15961 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:36011
I20260926 12:33:46.029209 16027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:36011 every 8 connection(s)
I20260926 12:33:46.030110 15961 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:33:46.033326 16028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:46.036398 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 15961
I20260926 12:33:46.036518 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:33:46.036778 10632 external_mini_cluster.cc:1428] Setting key b47690db0b86e411a2216d5c754fd5fc
I20260926 12:33:46.039161 16028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a: Bootstrap starting.
I20260926 12:33:46.041949 16028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:46.042802 16028 log.cc:824] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:46.044970 16028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a: No bootstrap required, opened a new log
I20260926 12:33:46.047451 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:46.038235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45312 (local address 127.10.98.62:36011)
0926 12:33:46.038935 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:46.038954 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:46.038997 (+ 43us) server_negotiation.cc:407] Connection header received
0926 12:33:46.039820 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:46.039845 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:46.040349 (+ 504us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:46.040738 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:46.041233 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:46.042678 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:46.043736 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:46.044458 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:46.044608 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:46.045424 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:46.045430 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:46.045447 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:33:46.045940 (+ 493us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:46.046011 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:33:46.046037 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:46.046133 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:46.046253 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 12:33:46.046523 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":113,"threads_started":1}
I20260926 12:33:46.049069 16028 raft_consensus.cc:348] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97e2c28ed1a46cf83689570b5dfc97a" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 36011 } }
I20260926 12:33:46.049366 16028 raft_consensus.cc:374] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:46.049425 16028 raft_consensus.cc:729] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e97e2c28ed1a46cf83689570b5dfc97a, State: Initialized, Role: FOLLOWER
I20260926 12:33:46.049847 16028 consensus_queue.cc:260] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97e2c28ed1a46cf83689570b5dfc97a" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 36011 } }
I20260926 12:33:46.049983 16028 raft_consensus.cc:388] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:33:46.050046 16028 raft_consensus.cc:482] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:33:46.050179 16028 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:46.050236 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:33:46.051168 16028 raft_consensus.cc:504] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97e2c28ed1a46cf83689570b5dfc97a" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 36011 } }
I20260926 12:33:46.051501 16028 leader_election.cc:302] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e97e2c28ed1a46cf83689570b5dfc97a; no voters:
I20260926 12:33:46.051797 16028 leader_election.cc:288] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:33:46.051926 16033 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:46.052169 16033 raft_consensus.cc:686] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a [term 1 LEADER]: Becoming Leader. State: Replica: e97e2c28ed1a46cf83689570b5dfc97a, State: Running, Role: LEADER
I20260926 12:33:46.052484 16033 consensus_queue.cc:237] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97e2c28ed1a46cf83689570b5dfc97a" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 36011 } }
I20260926 12:33:46.052990 16028 sys_catalog.cc:564] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:46.054133 16035 sys_catalog.cc:455] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e97e2c28ed1a46cf83689570b5dfc97a. Latest consensus state: current_term: 1 leader_uuid: "e97e2c28ed1a46cf83689570b5dfc97a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97e2c28ed1a46cf83689570b5dfc97a" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 36011 } } }
I20260926 12:33:46.054256 16035 sys_catalog.cc:458] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:46.054615 16034 sys_catalog.cc:455] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e97e2c28ed1a46cf83689570b5dfc97a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e97e2c28ed1a46cf83689570b5dfc97a" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 36011 } } }
I20260926 12:33:46.054728 16034 sys_catalog.cc:458] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:46.054633 16042 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:33:46.057813 16042 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:33:46.063241 16042 catalog_manager.cc:1347] Generated new cluster ID: d4d17bd3a1654cc8856e6c8bd9556ff2
I20260926 12:33:46.063354 16042 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:33:46.082650 16042 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:33:46.083846 16042 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:33:46.090159 16042 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e97e2c28ed1a46cf83689570b5dfc97a: Generated new TSK 0
I20260926 12:33:46.090881 16042 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:33:46.105170 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:46.100032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45316 (local address 127.10.98.62:36011)
0926 12:33:46.100156 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:46.100160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:46.100272 (+ 112us) server_negotiation.cc:407] Connection header received
0926 12:33:46.100427 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:46.100430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:46.100502 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:46.100619 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0926 12:33:46.100793 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:46.101675 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:46.102562 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:46.102906 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:46.103005 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:46.103281 (+ 276us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 12:33:46.104766 (+ 1485us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41571/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 12:33:46.104998 (+ 232us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45316: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41571/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 12:33:46.105245 16031 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45316: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41571/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 12:33:46.106869 16057 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:46.099942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:36011 (local address 127.0.0.1:45316)
0926 12:33:46.100192 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:46.100204 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:46.100310 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:46.100610 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:46.100612 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:46.100617 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0926 12:33:46.100707 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:46.100711 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:46.101797 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:46.101802 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:46.102450 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:46.102458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:46.103027 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:46.103030 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:46.103095 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:46.103118 (+ 23us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 12:33:46.104976 (+ 1858us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41571/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 12:33:46.106699 (+ 1723us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.98.62:36011: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41571/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":1538,"thread_start_us":78,"threads_started":1}
I20260926 12:33:46.107424 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 15961
E20260926 12:33:46.117762 16058 webserver.cc:533] Webserver: sslize failed:
E20260926 12:33:46.117848 16058 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T12:33:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (295 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T12:33:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:33:46Z Disabled control of system clock
I20260926 12:33:46.155444 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:36155
--webserver_interface=127.10.98.62
--webserver_port=0
--builtin_ntp_servers=127.10.98.20:43421
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:36155
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425997379116-10632-0/testchainca.cert with env {}
W20260926 12:33:46.266894 16062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:46.267192 16062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:46.267235 16062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:46.271397 16062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:46.271462 16062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:46.271477 16062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:46.271492 16062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:46.271507 16062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 12:33:46.271526 16062 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 12:33:46.271552 16062 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 12:33:46.277146 16062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:43421
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:36155
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:36155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425997379116-10632-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:46.278427 16062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:46.279636 16062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:46.288399 16068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:46.288444 16067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:46.288444 16070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:46.288548 16062 server_base.cc:1034] running on GCE node
I20260926 12:33:46.289288 16062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:46.290446 16062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:46.291615 16062 hybrid_clock.cc:648] HybridClock initialized: now 1790426026291600 us; error 30 us; skew 500 ppm
I20260926 12:33:46.293447 16062 webserver.cc:460] Webserver started at http://127.10.98.62:46463/ using document root <none> and password file <none>
I20260926 12:33:46.293994 16062 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:46.294059 16062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:46.294252 16062 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:46.296241 16062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "dbd29ba8d84c4499810563147a3fc8b8"
format_stamp: "Formatted at 2026-09-26 12:33:46 on dist-test-slave-t7tk"
server_key: "4cced178397caea0d0567494eb3354a0"
server_key_iv: "209939b40755d919139bd373a1db030d"
server_key_version: "encryptionkey@0"
I20260926 12:33:46.296762 16062 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "dbd29ba8d84c4499810563147a3fc8b8"
format_stamp: "Formatted at 2026-09-26 12:33:46 on dist-test-slave-t7tk"
server_key: "4cced178397caea0d0567494eb3354a0"
server_key_iv: "209939b40755d919139bd373a1db030d"
server_key_version: "encryptionkey@0"
I20260926 12:33:46.300292 16062 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 12:33:46.302673 16076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:46.303907 16062 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:33:46.304032 16062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "dbd29ba8d84c4499810563147a3fc8b8"
format_stamp: "Formatted at 2026-09-26 12:33:46 on dist-test-slave-t7tk"
server_key: "4cced178397caea0d0567494eb3354a0"
server_key_iv: "209939b40755d919139bd373a1db030d"
server_key_version: "encryptionkey@0"
I20260926 12:33:46.304128 16062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:46.320129 16062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:46.320873 16062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:46.321054 16062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:46.327586 16062 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:36155
I20260926 12:33:46.327610 16128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:36155 every 8 connection(s)
I20260926 12:33:46.328500 16062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:33:46.330737 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 16062
I20260926 12:33:46.330852 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:33:46.331141 10632 external_mini_cluster.cc:1428] Setting key 66e4fb521356848afa7c5ebec1197e8a
I20260926 12:33:46.331454 16129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:46.338006 16129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8: Bootstrap starting.
I20260926 12:33:46.340268 16129 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:46.340348 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:46.332489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33458 (local address 127.10.98.62:36155)
0926 12:33:46.333158 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:46.333173 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:46.333210 (+ 37us) server_negotiation.cc:407] Connection header received
0926 12:33:46.334133 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:46.334154 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:46.334562 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:46.334872 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:46.335299 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:46.336260 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:46.337156 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:46.337660 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:46.337807 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:46.338662 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:46.338668 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:46.338680 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:33:46.339072 (+ 392us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:46.339128 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:33:46.339147 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:46.339230 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:46.339369 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 12:33:46.339632 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":128,"threads_started":1}
I20260926 12:33:46.341174 16129 log.cc:824] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:46.343600 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:33:46.343755 16129 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8: No bootstrap required, opened a new log
I20260926 12:33:46.347368 16129 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd29ba8d84c4499810563147a3fc8b8" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 36155 } }
I20260926 12:33:46.347640 16129 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:46.347690 16129 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbd29ba8d84c4499810563147a3fc8b8, State: Initialized, Role: FOLLOWER
I20260926 12:33:46.348129 16129 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd29ba8d84c4499810563147a3fc8b8" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 36155 } }
I20260926 12:33:46.348255 16129 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:33:46.348313 16129 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:33:46.348412 16129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:46.349370 16129 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd29ba8d84c4499810563147a3fc8b8" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 36155 } }
I20260926 12:33:46.349709 16129 leader_election.cc:302] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbd29ba8d84c4499810563147a3fc8b8; no voters:
I20260926 12:33:46.350016 16129 leader_election.cc:288] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:33:46.350165 16134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:46.350404 16134 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8 [term 1 LEADER]: Becoming Leader. State: Replica: dbd29ba8d84c4499810563147a3fc8b8, State: Running, Role: LEADER
I20260926 12:33:46.350739 16134 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd29ba8d84c4499810563147a3fc8b8" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 36155 } }
I20260926 12:33:46.351279 16129 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:46.352639 16136 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbd29ba8d84c4499810563147a3fc8b8. Latest consensus state: current_term: 1 leader_uuid: "dbd29ba8d84c4499810563147a3fc8b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd29ba8d84c4499810563147a3fc8b8" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 36155 } } }
I20260926 12:33:46.352681 16135 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbd29ba8d84c4499810563147a3fc8b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbd29ba8d84c4499810563147a3fc8b8" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 36155 } } }
I20260926 12:33:46.352768 16136 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:46.352777 16135 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:46.353149 16146 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:33:46.355903 16146 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:33:46.361886 16146 catalog_manager.cc:1347] Generated new cluster ID: c74dcd5051e04e35be616a57d3007f1f
I20260926 12:33:46.361958 16146 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:33:46.376483 16146 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:33:46.377542 16146 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:33:46.382669 16146 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbd29ba8d84c4499810563147a3fc8b8: Generated new TSK 0
I20260926 12:33:46.383312 16146 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:33:46.479096 16132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:46.394611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.10.98.62:36155)
0926 12:33:46.394739 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:46.394743 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:46.394880 (+ 137us) server_negotiation.cc:407] Connection header received
0926 12:33:46.395022 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:46.395026 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:46.395080 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:46.395190 (+ 110us) server_negotiation.cc:226] Negotiated authn=JWT
0926 12:33:46.395407 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:46.396206 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:46.397053 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:46.397412 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:46.397502 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:46.397825 (+ 323us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 12:33:46.478591 (+ 80766us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 12:33:46.478940 (+ 349us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33464: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 12:33:46.479192 16132 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33464: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 12:33:46.479415 16158 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:46.394514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:36155 (local address 127.0.0.1:33464)
0926 12:33:46.394798 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:46.394811 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:46.394908 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:46.395200 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:46.395202 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:46.395208 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0926 12:33:46.395318 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:46.395322 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:46.396320 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:46.396323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:46.396949 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:46.396956 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:46.397566 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:46.397570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:46.397642 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:46.397664 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 12:33:46.478956 (+ 81292us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 12:33:46.479226 (+ 270us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.10.98.62:36155: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":96,"threads_started":1}
I20260926 12:33:46.480077 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 16062
2026-09-26T12:33:46Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (351 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (121 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (120 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:33:46 dist-test-slave-t7tk krb5kdc[16170](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:33:46 dist-test-slave-t7tk krb5kdc[16170](info): set up 2 sockets
Sep 26 12:33:46 dist-test-slave-t7tk krb5kdc[16170](info): commencing operation
krb5kdc: starting...
W20260926 12:33:47.678706 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:33:47 dist-test-slave-t7tk krb5kdc[16170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:33:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:33:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:33:47.796883 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:45301
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:45301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:47.903609 16183 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:47.903885 16183 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:47.903927 16183 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:47.908051 16183 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:47.908119 16183 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:47.908138 16183 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:47.908154 16183 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:47.908167 16183 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:33:47.913571 16183 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:45301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:45301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16183
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:47.914564 16183 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:47.915544 16183 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:47.923906 16189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:47.923894 16188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:47.923894 16191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:47.924009 16183 server_base.cc:1034] running on GCE node
I20260926 12:33:47.924690 16183 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:47.925769 16183 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:47.926927 16183 hybrid_clock.cc:648] HybridClock initialized: now 1790426027926910 us; error 31 us; skew 500 ppm
Sep 26 12:33:47 dist-test-slave-t7tk krb5kdc[16170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426027, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:47.929549 16183 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:33:47.930599 16183 webserver.cc:460] Webserver started at http://127.10.98.62:40245/ using document root <none> and password file <none>
I20260926 12:33:47.931135 16183 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:47.931201 16183 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:47.931387 16183 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:47.933503 16183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "b7ab8d18365744a981eba2d21b6a9132"
format_stamp: "Formatted at 2026-09-26 12:33:47 on dist-test-slave-t7tk"
server_key: "355727a606d881c4924a1b0d9d79be40"
server_key_iv: "4276b0aed488eab52c706963158deb06"
server_key_version: "encryptionkey@0"
I20260926 12:33:47.934053 16183 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "b7ab8d18365744a981eba2d21b6a9132"
format_stamp: "Formatted at 2026-09-26 12:33:47 on dist-test-slave-t7tk"
server_key: "355727a606d881c4924a1b0d9d79be40"
server_key_iv: "4276b0aed488eab52c706963158deb06"
server_key_version: "encryptionkey@0"
I20260926 12:33:47.937773 16183 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:33:47.940062 16198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:47.941346 16183 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:33:47.941473 16183 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "b7ab8d18365744a981eba2d21b6a9132"
format_stamp: "Formatted at 2026-09-26 12:33:47 on dist-test-slave-t7tk"
server_key: "355727a606d881c4924a1b0d9d79be40"
server_key_iv: "4276b0aed488eab52c706963158deb06"
server_key_version: "encryptionkey@0"
I20260926 12:33:47.941574 16183 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:47.958086 16183 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:47.960882 16183 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:47.961072 16183 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:47.968056 16183 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:45301
I20260926 12:33:47.968086 16250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:45301 every 8 connection(s)
I20260926 12:33:47.969036 16183 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:33:47.971758 16251 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:47.972712 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 16183
I20260926 12:33:47.972824 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:33:47.973091 10632 external_mini_cluster.cc:1428] Setting key 1f7d0d8c2cf2abeeb8603127b753946a
I20260926 12:33:47.977639 16251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132: Bootstrap starting.
I20260926 12:33:47.980089 16251 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:47.980780 16251 log.cc:824] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:33:47 dist-test-slave-t7tk krb5kdc[16170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:47.982786 16251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132: No bootstrap required, opened a new log
I20260926 12:33:47.986829 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:47.974431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41008 (local address 127.10.98.62:45301)
0926 12:33:47.974807 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:47.974822 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:47.974856 (+ 34us) server_negotiation.cc:407] Connection header received
0926 12:33:47.975462 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:47.975489 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:47.975812 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:47.976079 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:47.976921 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:47.977928 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:47.979046 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:47.979513 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:47.979625 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:47.982065 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:47.982071 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:47.982087 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:47.982117 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:47.983941 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:47.984333 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:47.984338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:47.984344 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:47.984475 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:47.984697 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:47.984700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:47.984701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:47.985249 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:47.985444 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:47.985632 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 12:33:47.985908 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":102,"threads_started":1}
I20260926 12:33:47.987201 16251 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ab8d18365744a981eba2d21b6a9132" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45301 } }
I20260926 12:33:47.987401 16251 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:47.987442 16251 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7ab8d18365744a981eba2d21b6a9132, State: Initialized, Role: FOLLOWER
I20260926 12:33:47.987836 16251 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ab8d18365744a981eba2d21b6a9132" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45301 } }
I20260926 12:33:47.987962 16251 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:33:47.988026 16251 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:33:47.988121 16251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:47.988922 16251 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ab8d18365744a981eba2d21b6a9132" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45301 } }
I20260926 12:33:47.989272 16251 leader_election.cc:302] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7ab8d18365744a981eba2d21b6a9132; no voters:
I20260926 12:33:47.989544 16251 leader_election.cc:288] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:33:47.989640 16256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:47.989845 16256 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132 [term 1 LEADER]: Becoming Leader. State: Replica: b7ab8d18365744a981eba2d21b6a9132, State: Running, Role: LEADER
I20260926 12:33:47.990222 16256 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ab8d18365744a981eba2d21b6a9132" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45301 } }
I20260926 12:33:47.990648 16251 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:47.991899 16257 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7ab8d18365744a981eba2d21b6a9132" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ab8d18365744a981eba2d21b6a9132" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45301 } } }
I20260926 12:33:47.991883 16258 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7ab8d18365744a981eba2d21b6a9132. Latest consensus state: current_term: 1 leader_uuid: "b7ab8d18365744a981eba2d21b6a9132" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7ab8d18365744a981eba2d21b6a9132" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45301 } } }
I20260926 12:33:47.992033 16257 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:47.992039 16258 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132 [sys.catalog]: This master's current role is: LEADER
W20260926 12:33:47.995535 16271 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:33:47.995600 16271 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:33:47.995677 16266 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:33:47.996557 16266 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:33:48.001929 16266 catalog_manager.cc:1347] Generated new cluster ID: 03d62964740740c5bc9c92c9f1f18a4b
I20260926 12:33:48.001998 16266 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:33:48.020884 16266 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:33:48.021847 16266 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:33:48.028405 16266 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7ab8d18365744a981eba2d21b6a9132: Generated new TSK 0
I20260926 12:33:48.029207 16266 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
I20260926 12:33:48.068094 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:45301
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:48.207772 16277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:48.208043 16277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:48.208084 16277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:48.212316 16277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:48.212392 16277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:48.212452 16277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:33:48.217860 16277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:45301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:48.219060 16277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:48.220158 16277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:48.229466 16283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:48.229610 16277 server_base.cc:1034] running on GCE node
W20260926 12:33:48.229458 16282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:48.229465 16285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:48.230165 16277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:48.230815 16277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:48.231979 16277 hybrid_clock.cc:648] HybridClock initialized: now 1790426028231960 us; error 32 us; skew 500 ppm
Sep 26 12:33:48 dist-test-slave-t7tk krb5kdc[16170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426028, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:48.234786 16277 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:33:48.235661 16277 webserver.cc:460] Webserver started at http://127.10.98.1:38783/ using document root <none> and password file <none>
I20260926 12:33:48.236191 16277 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:48.236250 16277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:48.236426 16277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:48.238484 16277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "70c9e953785743a685dd5786a552edff"
format_stamp: "Formatted at 2026-09-26 12:33:48 on dist-test-slave-t7tk"
server_key: "0e7d63b78cb3eef5339505621d3ed1da"
server_key_iv: "b58ed4b3911a5d22fe08c7686d0df030"
server_key_version: "encryptionkey@0"
I20260926 12:33:48.238993 16277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "70c9e953785743a685dd5786a552edff"
format_stamp: "Formatted at 2026-09-26 12:33:48 on dist-test-slave-t7tk"
server_key: "0e7d63b78cb3eef5339505621d3ed1da"
server_key_iv: "b58ed4b3911a5d22fe08c7686d0df030"
server_key_version: "encryptionkey@0"
I20260926 12:33:48.242620 16277 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:33:48.245031 16292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:48.246485 16277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:33:48.246601 16277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "70c9e953785743a685dd5786a552edff"
format_stamp: "Formatted at 2026-09-26 12:33:48 on dist-test-slave-t7tk"
server_key: "0e7d63b78cb3eef5339505621d3ed1da"
server_key_iv: "b58ed4b3911a5d22fe08c7686d0df030"
server_key_version: "encryptionkey@0"
I20260926 12:33:48.246711 16277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:48.259490 16277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:48.262741 16277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:48.262918 16277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:48.263509 16277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:48.264326 16277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:48.264372 16277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:48.264416 16277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:48.264444 16277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:48.274282 16277 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:43877
I20260926 12:33:48.274307 16405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:43877 every 8 connection(s)
I20260926 12:33:48.275242 16277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
Sep 26 12:33:48 dist-test-slave-t7tk krb5kdc[16170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426028, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:48.285449 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 16277
I20260926 12:33:48.285578 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:33:48.285895 10632 external_mini_cluster.cc:1428] Setting key 2457499da699c4df19bf2f483714fbf0
I20260926 12:33:48.290521 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:48.277177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:55969 (local address 127.10.98.62:45301)
0926 12:33:48.277317 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:48.277322 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:48.278268 (+ 946us) server_negotiation.cc:407] Connection header received
0926 12:33:48.279633 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:48.279636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:48.279684 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:48.279766 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:48.281048 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:48.281623 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:48.282789 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:48.283032 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:48.283126 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:48.285795 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:48.285798 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:48.285801 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:48.285906 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:48.288365 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:48.288853 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:48.288856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:48.288858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:48.288949 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:48.289330 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:48.289333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:48.289335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:48.289528 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:48.289615 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:48.290207 (+ 592us) server_negotiation.cc:299] Negotiation successful
0926 12:33:48.290354 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:33:48.291401 16408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:48.277641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45301 (local address 127.10.98.1:55969)
0926 12:33:48.278125 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:48.278168 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:48.279413 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:48.280021 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:48.280028 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:48.280396 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:48.280900 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:48.280914 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:48.281758 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:48.281761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:48.282675 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:48.282685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:48.283165 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:48.283167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:48.283309 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:48.283872 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:48.283888 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:48.285562 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:48.288480 (+ 2918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:48.288485 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:48.288499 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:48.288765 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:48.289024 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:48.289026 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:48.289027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:48.289231 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:48.289632 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:48.289641 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:48.289938 (+ 297us) client_negotiation.cc:769] Sending connection context
0926 12:33:48.290137 (+ 199us) client_negotiation.cc:240] Negotiation successful
0926 12:33:48.290490 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":130,"threads_started":1}
I20260926 12:33:48.292754 16406 heartbeater.cc:347] Connected to a master server at 127.10.98.62:45301
I20260926 12:33:48.292989 16406 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:48.293524 16406 heartbeater.cc:511] Master 127.10.98.62:45301 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.10.98.2@KRBTEST.COM; defaulting to no policy
I20260926 12:33:48.295188 16215 ts_manager.cc:194] Registered new tserver with Master: 70c9e953785743a685dd5786a552edff (127.10.98.1:43877)
I20260926 12:33:48.296326 16215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:55969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
I20260926 12:33:48.312300 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:45301
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:48.422024 16411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:48.422297 16411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:48.422340 16411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:48.426429 16411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:48.426494 16411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:48.426554 16411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:33:48.431875 16411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:45301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:48.432934 16411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:48.433977 16411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:48.443981 16419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:48.444057 16411 server_base.cc:1034] running on GCE node
W20260926 12:33:48.444005 16416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:48.443974 16417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:48.444707 16411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:48.445487 16411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:48.446671 16411 hybrid_clock.cc:648] HybridClock initialized: now 1790426028446644 us; error 38 us; skew 500 ppm
Sep 26 12:33:48 dist-test-slave-t7tk krb5kdc[16170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426028, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:48.449589 16411 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:33:48.450531 16411 webserver.cc:460] Webserver started at http://127.10.98.2:43961/ using document root <none> and password file <none>
I20260926 12:33:48.451099 16411 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:48.451167 16411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:48.451370 16411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:48.453516 16411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "d181f1484ca6436b8afacda8550e2b5f"
format_stamp: "Formatted at 2026-09-26 12:33:48 on dist-test-slave-t7tk"
server_key: "09f1fa65be4f899b0fe7eafd75295291"
server_key_iv: "d20c38a7818f87a28d060bbc169013c0"
server_key_version: "encryptionkey@0"
I20260926 12:33:48.454071 16411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "d181f1484ca6436b8afacda8550e2b5f"
format_stamp: "Formatted at 2026-09-26 12:33:48 on dist-test-slave-t7tk"
server_key: "09f1fa65be4f899b0fe7eafd75295291"
server_key_iv: "d20c38a7818f87a28d060bbc169013c0"
server_key_version: "encryptionkey@0"
I20260926 12:33:48.457805 16411 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:33:48.460242 16426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:48.461571 16411 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:33:48.461683 16411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "d181f1484ca6436b8afacda8550e2b5f"
format_stamp: "Formatted at 2026-09-26 12:33:48 on dist-test-slave-t7tk"
server_key: "09f1fa65be4f899b0fe7eafd75295291"
server_key_iv: "d20c38a7818f87a28d060bbc169013c0"
server_key_version: "encryptionkey@0"
I20260926 12:33:48.461786 16411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:48.475857 16411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:48.478972 16411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:48.479151 16411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:48.479727 16411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:48.480561 16411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:48.480607 16411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:48.480660 16411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:48.480685 16411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:48.489202 16411 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:35395
I20260926 12:33:48.489223 16539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:35395 every 8 connection(s)
I20260926 12:33:48.490092 16411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:33:48.498661 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 16411
I20260926 12:33:48.498769 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:33:48.499081 10632 external_mini_cluster.cc:1428] Setting key 23dbd04f9465a3b125cdc0d75f0378bb
Sep 26 12:33:48 dist-test-slave-t7tk krb5kdc[16170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426028, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:48.503736 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:48.491813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:40065 (local address 127.10.98.62:45301)
0926 12:33:48.491960 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:48.491965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:48.492886 (+ 921us) server_negotiation.cc:407] Connection header received
0926 12:33:48.494306 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:48.494310 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:48.494356 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:48.494431 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:48.495523 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:48.496076 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:48.497311 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:48.497530 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:48.497609 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:48.500096 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:48.500100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:48.500102 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:48.500137 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:48.501579 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:48.502241 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:48.502245 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:48.502247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:48.502345 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:48.502644 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:48.502647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:48.502648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:48.502858 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:48.502952 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:48.503439 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 12:33:48.503580 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:33:48.504593 16542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:48.492237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45301 (local address 127.10.98.2:40065)
0926 12:33:48.492743 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:48.492785 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:48.494106 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:48.494569 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:48.494575 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:48.494918 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:48.495383 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:48.495396 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:48.496176 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:48.496178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:48.497217 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:48.497226 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:48.497652 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:48.497655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:48.497799 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:48.498298 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:48.498313 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:48.499925 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:48.501690 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:48.501696 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:48.501712 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:48.502114 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:48.502431 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:48.502434 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:48.502435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:48.502569 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:48.502940 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:48.502945 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:48.503214 (+ 269us) client_negotiation.cc:769] Sending connection context
0926 12:33:48.503366 (+ 152us) client_negotiation.cc:240] Negotiation successful
0926 12:33:48.503617 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":158,"threads_started":1}
I20260926 12:33:48.505693 16540 heartbeater.cc:347] Connected to a master server at 127.10.98.62:45301
I20260926 12:33:48.505952 16540 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:48.506546 16540 heartbeater.cc:511] Master 127.10.98.62:45301 requested a full tablet report, sending...
I20260926 12:33:48.507654 16215 ts_manager.cc:194] Registered new tserver with Master: d181f1484ca6436b8afacda8550e2b5f (127.10.98.2:35395)
WARNING: no policy specified for kudu/127.10.98.3@KRBTEST.COM; defaulting to no policy
I20260926 12:33:48.508412 16215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:40065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
I20260926 12:33:48.528116 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:45301
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:35321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:48.637079 16545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:48.637426 16545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:48.637470 16545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:48.641611 16545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:48.641676 16545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:48.641747 16545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:33:48.647097 16545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:35321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:45301
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:48.648217 16545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:48.649297 16545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:48.659073 16553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:48.659174 16551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:48.659174 16550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:48.659436 16545 server_base.cc:1034] running on GCE node
I20260926 12:33:48.659868 16545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:48.660488 16545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:48.661670 16545 hybrid_clock.cc:648] HybridClock initialized: now 1790426028661648 us; error 35 us; skew 500 ppm
Sep 26 12:33:48 dist-test-slave-t7tk krb5kdc[16170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426028, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:48.664389 16545 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:33:48.665331 16545 webserver.cc:460] Webserver started at http://127.10.98.3:43029/ using document root <none> and password file <none>
I20260926 12:33:48.665890 16545 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:48.665963 16545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:48.666131 16545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:48.668128 16545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "b32604a644e9493aa92b60e9d60d2a73"
format_stamp: "Formatted at 2026-09-26 12:33:48 on dist-test-slave-t7tk"
server_key: "cd218282f925a8fb0af372fd08ab0933"
server_key_iv: "aed39b609aa5a1dc9312bc5c6f2c2ef6"
server_key_version: "encryptionkey@0"
I20260926 12:33:48.668646 16545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "b32604a644e9493aa92b60e9d60d2a73"
format_stamp: "Formatted at 2026-09-26 12:33:48 on dist-test-slave-t7tk"
server_key: "cd218282f925a8fb0af372fd08ab0933"
server_key_iv: "aed39b609aa5a1dc9312bc5c6f2c2ef6"
server_key_version: "encryptionkey@0"
I20260926 12:33:48.672359 16545 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 12:33:48.674768 16560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:48.676101 16545 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 12:33:48.676210 16545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "b32604a644e9493aa92b60e9d60d2a73"
format_stamp: "Formatted at 2026-09-26 12:33:48 on dist-test-slave-t7tk"
server_key: "cd218282f925a8fb0af372fd08ab0933"
server_key_iv: "aed39b609aa5a1dc9312bc5c6f2c2ef6"
server_key_version: "encryptionkey@0"
I20260926 12:33:48.676328 16545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:48.692312 16545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:48.695581 16545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:48.695755 16545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:48.696333 16545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:48.697183 16545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:48.697228 16545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:48.697271 16545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:48.697297 16545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:48.705698 16545 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:38325
I20260926 12:33:48.705724 16673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:38325 every 8 connection(s)
I20260926 12:33:48.706583 16545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:33:48.714326 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 16545
I20260926 12:33:48.714460 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:33:48.714789 10632 external_mini_cluster.cc:1428] Setting key e70ba8a8d30f82d120d958d722812319
Sep 26 12:33:48 dist-test-slave-t7tk krb5kdc[16170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426028, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:33:48.720068 16254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:48.708463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:50229 (local address 127.10.98.62:45301)
0926 12:33:48.708605 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:48.708609 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:48.709447 (+ 838us) server_negotiation.cc:407] Connection header received
0926 12:33:48.710795 (+ 1348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:48.710797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:48.710846 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:48.710921 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:48.712155 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:48.712711 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:48.713943 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:48.714163 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:48.714243 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:48.717151 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:48.717155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:48.717158 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:48.717187 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:48.718330 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:48.718799 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:48.718801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:48.718803 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:48.718871 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:48.719128 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:48.719130 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:48.719131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:48.719290 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:48.719357 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:48.719843 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 12:33:48.719945 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:33:48.720947 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:48.708814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45301 (local address 127.10.98.3:50229)
0926 12:33:48.709282 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:48.709334 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:48.710590 (+ 1256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:48.711145 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:48.711153 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:48.711516 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:48.711997 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:48.712011 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:48.712829 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:48.712832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:48.713827 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:48.713836 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:48.714316 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:48.714321 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:48.714550 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:48.715185 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:48.715206 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:48.716955 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:48.718460 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:48.718465 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:48.718477 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:48.718726 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:48.718934 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:48.718936 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:48.718938 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:48.719067 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:48.719371 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:48.719375 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:48.719610 (+ 235us) client_negotiation.cc:769] Sending connection context
0926 12:33:48.719801 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 12:33:48.720114 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":117,"threads_started":1}
I20260926 12:33:48.722033 16674 heartbeater.cc:347] Connected to a master server at 127.10.98.62:45301
I20260926 12:33:48.722262 16674 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:48.722740 16674 heartbeater.cc:511] Master 127.10.98.62:45301 requested a full tablet report, sending...
I20260926 12:33:48.723742 16215 ts_manager.cc:194] Registered new tserver with Master: b32604a644e9493aa92b60e9d60d2a73 (127.10.98.3:38325)
I20260926 12:33:48.724298 16215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:50229
I20260926 12:33:48.728946 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/test-admin.keytab.
Sep 26 12:33:48 dist-test-slave-t7tk krb5kdc[16170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:48.746322 10632 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 12:33:48.748970 16681 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:48.747642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.10.98.1:43877)
0926 12:33:48.747981 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:48.747997 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:48.748021 (+ 24us) server_negotiation.cc:407] Connection header received
0926 12:33:48.748073 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:48.748079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:48.748265 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:48.748383 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:48.748767 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48320: BlockingRecv error: recv got EOF from 127.0.0.1:48320 (error 108)
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":114,"threads_started":1}
W20260926 12:33:48.749215 10632 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/krb5cc)
Sep 26 12:33:48 dist-test-slave-t7tk krb5kdc[16170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:48.750216 10632 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 12:33:48 dist-test-slave-t7tk krb5kdc[16170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:48.751147 10632 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 12:33:48 dist-test-slave-t7tk krb5kdc[16170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.1@KRBTEST.COM
I20260926 12:33:48.760632 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:48.752066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48334 (local address 127.10.98.1:43877)
0926 12:33:48.752207 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:48.752212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:48.752287 (+ 75us) server_negotiation.cc:407] Connection header received
0926 12:33:48.752423 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:48.752425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:48.752466 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:48.752533 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:48.753142 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:48.753926 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:48.754751 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:48.755131 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:48.755224 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:48.757474 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:48.757484 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:48.757498 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:48.757524 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:48.758864 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:48.759302 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:48.759308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:48.759309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:48.759383 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:48.759657 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:48.759659 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:48.759660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:48.759941 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:48.760101 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:48.760302 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 12:33:48.760438 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260926 12:33:48.764542 16681 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:48.763635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48342 (local address 127.10.98.1:43877)
0926 12:33:48.763767 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:48.763770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:48.763873 (+ 103us) server_negotiation.cc:407] Connection header received
0926 12:33:48.763949 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:48.763951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:48.763989 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:48.764050 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:48.764411 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48342: BlockingRecv error: recv got EOF from 127.0.0.1:48342 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260926 12:33:48.764618 10632 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/krb5cc' of type FILE
Sep 26 12:33:48 dist-test-slave-t7tk krb5kdc[16170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:48.765722 10632 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 12:33:48 dist-test-slave-t7tk krb5kdc[16170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:48.766534 10632 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 12:33:48 dist-test-slave-t7tk krb5kdc[16170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.1@KRBTEST.COM
I20260926 12:33:48.774930 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:48.767514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48352 (local address 127.10.98.1:43877)
0926 12:33:48.767624 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:48.767627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:48.767640 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:33:48.767751 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:48.767754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:48.767789 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:48.767865 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:48.768388 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:48.768958 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:48.769795 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:48.770037 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:48.770115 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:48.772110 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:48.772113 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:48.772115 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:48.772138 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:48.773565 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:48.773935 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:48.773937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:48.773938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:48.774008 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:48.774194 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:48.774196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:48.774197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:48.774344 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:48.774436 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:48.774680 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 12:33:48.774773 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 12:33:48.777583 16681 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:48.776643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48354 (local address 127.10.98.1:43877)
0926 12:33:48.776760 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:48.776764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:48.776840 (+ 76us) server_negotiation.cc:407] Connection header received
0926 12:33:48.776952 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:48.776955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:48.777003 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:48.777064 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:48.777497 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48354: BlockingRecv error: recv got EOF from 127.0.0.1:48354 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260926 12:33:48.777817 10632 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790425997379116-10632-0/krb5kdc/krb5cc' of type FILE
Sep 26 12:33:48 dist-test-slave-t7tk krb5kdc[16170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:48.778774 10632 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 12:33:48 dist-test-slave-t7tk krb5kdc[16170](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:48.779644 10632 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 12:33:48 dist-test-slave-t7tk krb5kdc[16170](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.1@KRBTEST.COM
I20260926 12:33:48.787811 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:48.780537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48360 (local address 127.10.98.1:43877)
0926 12:33:48.780637 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:48.780641 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:48.780716 (+ 75us) server_negotiation.cc:407] Connection header received
0926 12:33:48.780811 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:48.780814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:48.780880 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:48.780943 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:48.781496 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:48.782009 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:48.782761 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:48.782992 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:48.783077 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:48.785092 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:48.785123 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:48.785125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:48.785151 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:48.786328 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:48.786721 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:48.786724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:48.786725 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:48.786826 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:48.787087 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:48.787090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:48.787092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:48.787250 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:48.787359 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:48.787572 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 12:33:48.787670 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 12:33:48.788736 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 16277
I20260926 12:33:48.794266 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 16411
I20260926 12:33:48.799105 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 16545
I20260926 12:33:48.803442 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 16183
2026-09-26T12:33:48Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2083 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:33:48 dist-test-slave-t7tk krb5kdc[16694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:33:48 dist-test-slave-t7tk krb5kdc[16694](info): set up 2 sockets
Sep 26 12:33:48 dist-test-slave-t7tk krb5kdc[16694](info): commencing operation
krb5kdc: starting...
W20260926 12:33:49.766891 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:33:49 dist-test-slave-t7tk krb5kdc[16694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:33:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:33:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.62.keytab.
Entry for principal oryx/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.62.keytab.
I20260926 12:33:49.889467 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:40045
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.62.keytab
--principal=oryx/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:40045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:49.999432 16707 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:49.999734 16707 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:49.999794 16707 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:50.003885 16707 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:50.003957 16707 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:50.003985 16707 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:50.004006 16707 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:50.004027 16707 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:33:50.009743 16707 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:40045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.62.keytab
--principal=oryx/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:40045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16707
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:50.010859 16707 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:50.011902 16707 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:50.021055 16713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:50.021291 16707 server_base.cc:1034] running on GCE node
W20260926 12:33:50.021191 16715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:50.021047 16712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:50.022035 16707 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:50.023280 16707 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:50.024487 16707 hybrid_clock.cc:648] HybridClock initialized: now 1790426030024461 us; error 48 us; skew 500 ppm
Sep 26 12:33:50 dist-test-slave-t7tk krb5kdc[16694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426030, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:50.027561 16707 init.cc:377] Logged in from keytab as oryx/127.10.98.62@KRBTEST.COM (short username oryx)
I20260926 12:33:50.028548 16707 webserver.cc:460] Webserver started at http://127.10.98.62:39637/ using document root <none> and password file <none>
I20260926 12:33:50.029258 16707 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:50.029338 16707 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:50.029573 16707 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:50.031675 16707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "e1e1ab3339074143aa34a2cbe581dc5b"
format_stamp: "Formatted at 2026-09-26 12:33:50 on dist-test-slave-t7tk"
server_key: "400e0264a6b04ac1ab46e1400ed77deb"
server_key_iv: "ec757587206edfdd3ce927625ff9083a"
server_key_version: "encryptionkey@0"
I20260926 12:33:50.032224 16707 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "e1e1ab3339074143aa34a2cbe581dc5b"
format_stamp: "Formatted at 2026-09-26 12:33:50 on dist-test-slave-t7tk"
server_key: "400e0264a6b04ac1ab46e1400ed77deb"
server_key_iv: "ec757587206edfdd3ce927625ff9083a"
server_key_version: "encryptionkey@0"
I20260926 12:33:50.036172 16707 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:33:50.038856 16722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:50.040301 16707 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:33:50.040429 16707 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "e1e1ab3339074143aa34a2cbe581dc5b"
format_stamp: "Formatted at 2026-09-26 12:33:50 on dist-test-slave-t7tk"
server_key: "400e0264a6b04ac1ab46e1400ed77deb"
server_key_iv: "ec757587206edfdd3ce927625ff9083a"
server_key_version: "encryptionkey@0"
I20260926 12:33:50.040557 16707 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:50.056717 16707 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:50.059851 16707 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:50.060037 16707 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:50.066954 16707 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:40045
I20260926 12:33:50.067252 16774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:40045 every 8 connection(s)
I20260926 12:33:50.067931 16707 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:33:50.070770 16775 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:50.075929 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 16707
I20260926 12:33:50.076066 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:33:50.076399 10632 external_mini_cluster.cc:1428] Setting key 6a24284e8c9a60eb816ccb6a24fd57c1
I20260926 12:33:50.076393 16775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b: Bootstrap starting.
I20260926 12:33:50.078920 16775 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:50.079686 16775 log.cc:824] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:50.081708 16775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b: No bootstrap required, opened a new log
Sep 26 12:33:50 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM
I20260926 12:33:50.086038 16775 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } }
I20260926 12:33:50.086311 16775 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:50.086351 16775 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1e1ab3339074143aa34a2cbe581dc5b, State: Initialized, Role: FOLLOWER
I20260926 12:33:50.086849 16775 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } }
I20260926 12:33:50.086990 16775 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:33:50.087056 16775 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:33:50.087194 16775 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:50.088423 16775 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } }
I20260926 12:33:50.088775 16775 leader_election.cc:302] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1e1ab3339074143aa34a2cbe581dc5b; no voters:
I20260926 12:33:50.089172 16775 leader_election.cc:288] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:33:50.089267 16780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:50.089470 16780 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [term 1 LEADER]: Becoming Leader. State: Replica: e1e1ab3339074143aa34a2cbe581dc5b, State: Running, Role: LEADER
I20260926 12:33:50.089829 16780 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } }
I20260926 12:33:50.090584 16775 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:50.091529 16782 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1e1ab3339074143aa34a2cbe581dc5b. Latest consensus state: current_term: 1 leader_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } }
I20260926 12:33:50.091678 16782 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:50.091533 16781 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } }
I20260926 12:33:50.091792 16781 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:50.092332 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:50.077771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53650 (local address 127.10.98.62:40045)
0926 12:33:50.078206 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:50.078225 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:50.078291 (+ 66us) server_negotiation.cc:407] Connection header received
0926 12:33:50.078976 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:50.078999 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:50.079349 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:50.079634 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:50.080468 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:50.081605 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:50.082746 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:50.083242 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:50.083359 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:50.086217 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:50.086223 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:50.086236 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:50.086266 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:50.088271 (+ 2005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:50.088741 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:50.088747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:50.088753 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:50.088930 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:50.089298 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:50.089301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:50.089302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:50.089824 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:50.090015 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:50.090868 (+ 853us) server_negotiation.cc:299] Negotiation successful
0926 12:33:50.091183 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":116,"threads_started":1}
W20260926 12:33:50.095786 16795 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:33:50.095854 16795 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:33:50.095944 16789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:33:50.096643 16789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:33:50.101827 16789 catalog_manager.cc:1347] Generated new cluster ID: f47659f98ed746639f42fd35d409569e
I20260926 12:33:50.101902 16789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:33:50.117664 16789 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:33:50.118647 16789 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:33:50.126814 16789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b: Generated new TSK 0
I20260926 12:33:50.127686 16789 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.1.keytab.
Entry for principal oryx/127.10.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.1.keytab.
I20260926 12:33:50.161283 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:40045
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.1.keytab
--principal=oryx/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:50.291287 16801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:50.291648 16801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:50.291706 16801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:50.297271 16801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:50.297392 16801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:50.297493 16801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:33:50.303896 16801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.1.keytab
--principal=oryx/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:40045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:50.304957 16801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:50.306005 16801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:50.315608 16807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:50.315732 16806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:50.315603 16809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:50.315722 16801 server_base.cc:1034] running on GCE node
I20260926 12:33:50.316211 16801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:50.316816 16801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:50.317978 16801 hybrid_clock.cc:648] HybridClock initialized: now 1790426030317935 us; error 51 us; skew 500 ppm
Sep 26 12:33:50 dist-test-slave-t7tk krb5kdc[16694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426030, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:50.320521 16801 init.cc:377] Logged in from keytab as oryx/127.10.98.1@KRBTEST.COM (short username oryx)
I20260926 12:33:50.321409 16801 webserver.cc:460] Webserver started at http://127.10.98.1:46523/ using document root <none> and password file <none>
I20260926 12:33:50.321939 16801 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:50.322006 16801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:50.322197 16801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:50.324280 16801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "d4f592a9801e467da0bc143fe5c41cbc"
format_stamp: "Formatted at 2026-09-26 12:33:50 on dist-test-slave-t7tk"
server_key: "8faafd2e8be705fda2c3447c66358528"
server_key_iv: "eb2f7e368507479fc4c3e819747dcfbe"
server_key_version: "encryptionkey@0"
I20260926 12:33:50.324836 16801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "d4f592a9801e467da0bc143fe5c41cbc"
format_stamp: "Formatted at 2026-09-26 12:33:50 on dist-test-slave-t7tk"
server_key: "8faafd2e8be705fda2c3447c66358528"
server_key_iv: "eb2f7e368507479fc4c3e819747dcfbe"
server_key_version: "encryptionkey@0"
I20260926 12:33:50.328567 16801 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 12:33:50.331166 16816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:50.332444 16801 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:33:50.332554 16801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "d4f592a9801e467da0bc143fe5c41cbc"
format_stamp: "Formatted at 2026-09-26 12:33:50 on dist-test-slave-t7tk"
server_key: "8faafd2e8be705fda2c3447c66358528"
server_key_iv: "eb2f7e368507479fc4c3e819747dcfbe"
server_key_version: "encryptionkey@0"
I20260926 12:33:50.332654 16801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:50.349275 16801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:50.352837 16801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:50.353060 16801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:50.353744 16801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:50.354660 16801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:50.354713 16801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:50.354771 16801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:50.354805 16801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:50.364521 16801 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:45527
I20260926 12:33:50.364607 16929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:45527 every 8 connection(s)
I20260926 12:33:50.365554 16801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:33:50.370096 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 16801
I20260926 12:33:50.370225 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:33:50.370532 10632 external_mini_cluster.cc:1428] Setting key a580d704a1cd2fd788e96e564c1faf02
Sep 26 12:33:50 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426030, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.10.98.2@KRBTEST.COM; defaulting to no policy
I20260926 12:33:50.380119 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:50.367545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:33339 (local address 127.10.98.62:40045)
0926 12:33:50.367699 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:50.367703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:50.368604 (+ 901us) server_negotiation.cc:407] Connection header received
0926 12:33:50.370034 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:50.370043 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:50.370109 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:50.370193 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:50.371710 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:50.372227 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:50.373395 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:50.373621 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:50.373709 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:50.376374 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:50.376384 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:50.376387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:50.376417 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:50.378108 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:50.378624 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:50.378627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:50.378628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:50.378701 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:50.379020 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:50.379024 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:50.379025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:50.379206 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:50.379289 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:50.379826 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 12:33:50.379955 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:33:50.381016 16932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:50.367930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.1:33339)
0926 12:33:50.368438 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:50.368484 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:50.369806 (+ 1322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:50.370438 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:50.370477 (+ 39us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:50.370982 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:50.371565 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:50.371583 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:50.372347 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:50.372350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:50.373285 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:50.373294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:50.373759 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:50.373763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:50.373916 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:50.374540 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:50.374557 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:50.376096 (+ 1539us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:50.378232 (+ 2136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:50.378238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:50.378257 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:50.378528 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:50.378784 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:50.378788 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:50.378790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:50.378934 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:50.379301 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:50.379308 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:50.379596 (+ 288us) client_negotiation.cc:769] Sending connection context
0926 12:33:50.379776 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 12:33:50.380115 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":113,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.98.2@KRBTEST.COM" created.
I20260926 12:33:50.382305 16930 heartbeater.cc:347] Connected to a master server at 127.10.98.62:40045
I20260926 12:33:50.382545 16930 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:50.383039 16930 heartbeater.cc:511] Master 127.10.98.62:40045 requested a full tablet report, sending...
I20260926 12:33:50.384712 16739 ts_manager.cc:194] Registered new tserver with Master: d4f592a9801e467da0bc143fe5c41cbc (127.10.98.1:45527)
I20260926 12:33:50.385905 16739 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:33339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.2.keytab.
Entry for principal oryx/127.10.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.2.keytab.
I20260926 12:33:50.400013 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:40045
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.2.keytab
--principal=oryx/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:50.515744 16935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:50.516012 16935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:50.516073 16935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:50.520268 16935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:50.520344 16935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:50.520427 16935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:33:50.526041 16935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.2.keytab
--principal=oryx/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:40045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:50.527467 16935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:50.528650 16935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:50.538540 16940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:50.538668 16943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:50.538618 16941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:50.538704 16935 server_base.cc:1034] running on GCE node
I20260926 12:33:50.539167 16935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:50.539834 16935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:50.541090 16935 hybrid_clock.cc:648] HybridClock initialized: now 1790426030541069 us; error 32 us; skew 500 ppm
Sep 26 12:33:50 dist-test-slave-t7tk krb5kdc[16694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426030, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:50.544044 16935 init.cc:377] Logged in from keytab as oryx/127.10.98.2@KRBTEST.COM (short username oryx)
I20260926 12:33:50.545010 16935 webserver.cc:460] Webserver started at http://127.10.98.2:41627/ using document root <none> and password file <none>
I20260926 12:33:50.545594 16935 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:50.545675 16935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:50.545898 16935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:50.548013 16935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "08fc683fa8894fa685caba488359483d"
format_stamp: "Formatted at 2026-09-26 12:33:50 on dist-test-slave-t7tk"
server_key: "d9c4eed500a21070027c5583819e04d2"
server_key_iv: "49e5bf16dd10954f932b8d54d0d5912b"
server_key_version: "encryptionkey@0"
I20260926 12:33:50.548550 16935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "08fc683fa8894fa685caba488359483d"
format_stamp: "Formatted at 2026-09-26 12:33:50 on dist-test-slave-t7tk"
server_key: "d9c4eed500a21070027c5583819e04d2"
server_key_iv: "49e5bf16dd10954f932b8d54d0d5912b"
server_key_version: "encryptionkey@0"
I20260926 12:33:50.552476 16935 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:33:50.555202 16950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:50.556630 16935 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260926 12:33:50.556757 16935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "08fc683fa8894fa685caba488359483d"
format_stamp: "Formatted at 2026-09-26 12:33:50 on dist-test-slave-t7tk"
server_key: "d9c4eed500a21070027c5583819e04d2"
server_key_iv: "49e5bf16dd10954f932b8d54d0d5912b"
server_key_version: "encryptionkey@0"
I20260926 12:33:50.556869 16935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:50.590934 16935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:50.594197 16935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:50.594373 16935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:50.594951 16935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:50.595769 16935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:50.595817 16935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:50.595870 16935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:50.595896 16935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:50.605057 16935 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:33289
I20260926 12:33:50.605239 17063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:33289 every 8 connection(s)
I20260926 12:33:50.606014 16935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:33:50.606943 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 16935
I20260926 12:33:50.607039 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:33:50.607287 10632 external_mini_cluster.cc:1428] Setting key f3eec4ff2a883a5a28567fa9abb42ef8
WARNING: no policy specified for oryx/127.10.98.3@KRBTEST.COM; defaulting to no policy
Sep 26 12:33:50 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426030, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.98.3@KRBTEST.COM" created.
I20260926 12:33:50.621228 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:50.608010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:54847 (local address 127.10.98.62:40045)
0926 12:33:50.608122 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:50.608125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:50.609171 (+ 1046us) server_negotiation.cc:407] Connection header received
0926 12:33:50.610617 (+ 1446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:50.610621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:50.610678 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:50.610759 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:50.611917 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:50.612711 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:50.614273 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:50.614615 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:50.614738 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:50.617436 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:50.617439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:50.617442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:50.617474 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:50.618866 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:50.619448 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:50.619451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:50.619452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:50.619527 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:50.619923 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:50.619926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:50.619927 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:50.620124 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:50.620207 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:50.620891 (+ 684us) server_negotiation.cc:299] Negotiation successful
0926 12:33:50.621025 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 12:33:50.622041 17066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:50.608470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.2:54847)
0926 12:33:50.608993 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:50.609048 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:50.610399 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:50.610909 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:50.610916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:50.611278 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:50.611774 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:50.611789 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:50.612834 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:50.612837 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:50.614128 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:50.614139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:50.614808 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:50.614811 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:50.615030 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:50.615611 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:50.615627 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:50.617229 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:50.619032 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:50.619037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:50.619050 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:50.619334 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:50.619613 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:50.619616 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:50.619618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:50.619794 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:50.620218 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:50.620225 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:50.620550 (+ 325us) client_negotiation.cc:769] Sending connection context
0926 12:33:50.620769 (+ 219us) client_negotiation.cc:240] Negotiation successful
0926 12:33:50.621045 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":128,"threads_started":1}
I20260926 12:33:50.623574 17064 heartbeater.cc:347] Connected to a master server at 127.10.98.62:40045
I20260926 12:33:50.623826 17064 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:50.624351 17064 heartbeater.cc:511] Master 127.10.98.62:40045 requested a full tablet report, sending...
I20260926 12:33:50.625525 16739 ts_manager.cc:194] Registered new tserver with Master: 08fc683fa8894fa685caba488359483d (127.10.98.2:33289)
I20260926 12:33:50.626302 16739 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:54847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.3.keytab.
Entry for principal oryx/127.10.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.3.keytab.
I20260926 12:33:50.637282 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:40045
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.3.keytab
--principal=oryx/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:50.745913 17069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:50.746184 17069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:50.746243 17069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:50.750276 17069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:50.750344 17069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:50.750427 17069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:33:50.755764 17069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.3.keytab
--principal=oryx/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:40045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17069
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:50.756831 17069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:50.757871 17069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:50.767428 17077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:50.767447 17075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:50.767426 17074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:50.767684 17069 server_base.cc:1034] running on GCE node
I20260926 12:33:50.768190 17069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:50.768800 17069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:50.769953 17069 hybrid_clock.cc:648] HybridClock initialized: now 1790426030769927 us; error 41 us; skew 500 ppm
Sep 26 12:33:50 dist-test-slave-t7tk krb5kdc[16694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426030, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:50.772585 17069 init.cc:377] Logged in from keytab as oryx/127.10.98.3@KRBTEST.COM (short username oryx)
I20260926 12:33:50.773571 17069 webserver.cc:460] Webserver started at http://127.10.98.3:38669/ using document root <none> and password file <none>
I20260926 12:33:50.774113 17069 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:50.774178 17069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:50.774371 17069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:50.776418 17069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "46015c6e40794d6486049f51816f2200"
format_stamp: "Formatted at 2026-09-26 12:33:50 on dist-test-slave-t7tk"
server_key: "113cfb34f2dba6d61572d21c278ffe71"
server_key_iv: "8b1c24061c6dda1a402d8aaa0adb5572"
server_key_version: "encryptionkey@0"
I20260926 12:33:50.776953 17069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "46015c6e40794d6486049f51816f2200"
format_stamp: "Formatted at 2026-09-26 12:33:50 on dist-test-slave-t7tk"
server_key: "113cfb34f2dba6d61572d21c278ffe71"
server_key_iv: "8b1c24061c6dda1a402d8aaa0adb5572"
server_key_version: "encryptionkey@0"
I20260926 12:33:50.780529 17069 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:33:50.782945 17084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:50.784148 17069 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 12:33:50.784258 17069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "46015c6e40794d6486049f51816f2200"
format_stamp: "Formatted at 2026-09-26 12:33:50 on dist-test-slave-t7tk"
server_key: "113cfb34f2dba6d61572d21c278ffe71"
server_key_iv: "8b1c24061c6dda1a402d8aaa0adb5572"
server_key_version: "encryptionkey@0"
I20260926 12:33:50.784350 17069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:50.799201 17069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:50.802371 17069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:50.802547 17069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:50.803139 17069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:50.803931 17069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:50.803978 17069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:50.804041 17069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:50.804068 17069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:50.812332 17069 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:46769
I20260926 12:33:50.812362 17197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:46769 every 8 connection(s)
I20260926 12:33:50.813244 17069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
Sep 26 12:33:50 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426030, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM
I20260926 12:33:50.823400 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 17069
I20260926 12:33:50.823525 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:33:50.823786 10632 external_mini_cluster.cc:1428] Setting key 3b16d11ed8f18cfc3f58f8360da5d45b
I20260926 12:33:50.826254 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:50.815093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:36197 (local address 127.10.98.62:40045)
0926 12:33:50.815240 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:50.815244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:50.816103 (+ 859us) server_negotiation.cc:407] Connection header received
0926 12:33:50.817367 (+ 1264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:50.817370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:50.817416 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:50.817486 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:50.818571 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:50.819087 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:50.820200 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:50.820403 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:50.820472 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:50.822810 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:50.822813 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:50.822815 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:50.822839 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:50.824203 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:50.824735 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:50.824738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:50.824740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:50.824815 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:50.825134 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:50.825138 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:50.825140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:50.825322 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:50.825405 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:50.825965 (+ 560us) server_negotiation.cc:299] Negotiation successful
0926 12:33:50.826101 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:33:50.827040 17200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:50.815464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.3:36197)
0926 12:33:50.815903 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:50.815945 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:50.817191 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:50.817642 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:50.817649 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:50.818003 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:50.818462 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:50.818475 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:50.819189 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:50.819192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:50.820108 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:50.820117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:50.820510 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:50.820512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:50.820661 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:50.821181 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:50.821196 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:50.822650 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:50.824303 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:50.824309 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:50.824324 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:50.824651 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:50.824894 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:50.824897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:50.824899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:50.825031 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:50.825418 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:50.825425 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:50.825738 (+ 313us) client_negotiation.cc:769] Sending connection context
0926 12:33:50.825912 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 12:33:50.826232 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":105,"threads_started":1}
I20260926 12:33:50.828102 17198 heartbeater.cc:347] Connected to a master server at 127.10.98.62:40045
I20260926 12:33:50.828351 17198 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:50.828866 17198 heartbeater.cc:511] Master 127.10.98.62:40045 requested a full tablet report, sending...
I20260926 12:33:50.829923 16739 ts_manager.cc:194] Registered new tserver with Master: 46015c6e40794d6486049f51816f2200 (127.10.98.3:46769)
I20260926 12:33:50.830487 16739 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:36197
I20260926 12:33:50.837585 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.61.keytab.
Entry for principal oryx/127.10.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.61.keytab.
I20260926 12:33:50.864544 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.61:38279
--webserver_interface=127.10.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.61.keytab
--principal=oryx/127.10.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:40045,127.10.98.61:38279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:50.973361 17203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:50.973634 17203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:50.973673 17203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:50.977681 17203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:50.977743 17203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:50.977762 17203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:50.977777 17203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:50.977790 17203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:33:50.983022 17203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:40045,127.10.98.61:38279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.61.keytab
--principal=oryx/127.10.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.61:38279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
W20260926 12:33:50.983225 17203 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.10.98.62:40045,127.10.98.61:38279'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 12:33:50.984061 17203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:50.985018 17203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:33:50.993420 17203 server_base.cc:1034] running on GCE node
W20260926 12:33:50.993413 17208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:50.993479 17209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:50.993438 17211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:50.994238 17203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:50.995369 17203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:50.996536 17203 hybrid_clock.cc:648] HybridClock initialized: now 1790426030996515 us; error 35 us; skew 500 ppm
Sep 26 12:33:50 dist-test-slave-t7tk krb5kdc[16694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426030, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:50.999186 17203 init.cc:377] Logged in from keytab as oryx/127.10.98.61@KRBTEST.COM (short username oryx)
I20260926 12:33:51.000048 17203 webserver.cc:460] Webserver started at http://127.10.98.61:37983/ using document root <none> and password file <none>
I20260926 12:33:51.000552 17203 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:51.000622 17203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:51.000869 17203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:51.002882 17203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/data/instance:
uuid: "08b7010c899c498e9e6af4db7d5e95d2"
format_stamp: "Formatted at 2026-09-26 12:33:51 on dist-test-slave-t7tk"
server_key: "02cd0ea6018e414f6c51dba83f8a14e3"
server_key_iv: "8d6406bd9b23dc68133c9bf3266602bd"
server_key_version: "encryptionkey@0"
I20260926 12:33:51.003371 17203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/wal/instance:
uuid: "08b7010c899c498e9e6af4db7d5e95d2"
format_stamp: "Formatted at 2026-09-26 12:33:51 on dist-test-slave-t7tk"
server_key: "02cd0ea6018e414f6c51dba83f8a14e3"
server_key_iv: "8d6406bd9b23dc68133c9bf3266602bd"
server_key_version: "encryptionkey@0"
I20260926 12:33:51.006987 17203 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:33:51.009361 17218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:51.010672 17203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 12:33:51.010782 17203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/wal
uuid: "08b7010c899c498e9e6af4db7d5e95d2"
format_stamp: "Formatted at 2026-09-26 12:33:51 on dist-test-slave-t7tk"
server_key: "02cd0ea6018e414f6c51dba83f8a14e3"
server_key_iv: "8d6406bd9b23dc68133c9bf3266602bd"
server_key_version: "encryptionkey@0"
I20260926 12:33:51.010885 17203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:51.020702 17203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:51.023870 17203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:51.024029 17203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:51.030844 17203 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.61:38279
I20260926 12:33:51.031172 17270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.61:38279 every 8 connection(s)
I20260926 12:33:51.031826 17203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/data/info.pb
I20260926 12:33:51.034711 17271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:51.038512 17271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:33:51.040443 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 17203
Sep 26 12:33:51 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426030, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.61@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM
Sep 26 12:33:51 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.98.61@KRBTEST.COM
I20260926 12:33:51.051846 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.040367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53662 (local address 127.10.98.62:40045)
0926 12:33:51.040517 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.040522 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.041713 (+ 1191us) server_negotiation.cc:407] Connection header received
0926 12:33:51.042720 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.042724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.042779 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.042858 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.043749 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.044296 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.045540 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.045754 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.045842 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.048355 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.048358 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.048360 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.048386 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.049860 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.050299 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.050302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.050304 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.050383 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.050688 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.050691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.050692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.050912 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.050998 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.051570 (+ 572us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.051688 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:33:51.052701 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.040911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.0.0.1:53662)
0926 12:33:51.041552 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.041609 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.042491 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.042930 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.042936 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.043298 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:51.043601 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.043621 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.044455 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.044457 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.045420 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.045431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.045903 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.045906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.046023 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.046615 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:51.046630 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:51.048207 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:51.049961 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.049969 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.049980 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.050207 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.050463 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.050466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.050468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.050601 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.051017 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:51.051024 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:51.051352 (+ 328us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.051505 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.052023 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":303,"thread_start_us":177,"threads_started":1}
I20260926 12:33:51.052706 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.041297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49806 (local address 127.10.98.61:38279)
0926 12:33:51.041684 (+ 387us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.041694 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.041732 (+ 38us) server_negotiation.cc:407] Connection header received
0926 12:33:51.042452 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.042468 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.042668 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.042834 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.043639 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.044518 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.045433 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.045796 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.045937 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.048673 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.048678 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.048688 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.048708 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.050064 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.050443 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.050448 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.050449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.050583 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.050823 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.050826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.050828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.051275 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.051444 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.051661 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.051823 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":303,"thread_start_us":120,"threads_started":1}
I20260926 12:33:51.054116 17271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:33:51 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426030, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.61@KRBTEST.COM for oryx/127.10.98.61@KRBTEST.COM
I20260926 12:33:51.062273 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.054716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49822 (local address 127.10.98.61:38279)
0926 12:33:51.054949 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.054983 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.054995 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:33:51.055023 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.055025 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.055057 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.055120 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.055457 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.055949 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.056643 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.057346 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.057436 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.059375 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.059379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.059381 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.059400 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.060657 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.061130 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.061133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.061135 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.061203 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.061463 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.061466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.061468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.061672 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.061755 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.062062 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.062151 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":169}
I20260926 12:33:51.062273 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.054579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:38279 (local address 127.0.0.1:49822)
0926 12:33:51.054676 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.054690 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.054755 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.055144 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.055147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.055284 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:51.055364 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.055369 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.056038 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.056042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.056548 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.056555 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.057444 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.057447 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.057538 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.058056 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:51.058067 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:51.059257 (+ 1190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:51.060763 (+ 1506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.060765 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.060767 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.061028 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.061277 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.061280 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.061281 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.061373 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.061755 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:51.061758 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:51.061860 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.061945 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.062079 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":1}
I20260926 12:33:51.065512 17271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: Bootstrap starting.
I20260926 12:33:51.067772 17271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:51.068483 17271 log.cc:824] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:51.070525 17271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: No bootstrap required, opened a new log
I20260926 12:33:51.073493 17271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } }
I20260926 12:33:51.073705 17271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:51.073741 17271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08b7010c899c498e9e6af4db7d5e95d2, State: Initialized, Role: FOLLOWER
I20260926 12:33:51.074183 17271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } }
I20260926 12:33:51.074663 17277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } } }
I20260926 12:33:51.074793 17277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:33:51.075326 17271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 12:33:51.079931 17288 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:33:51.080019 17288 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:33:51.081404 17203 master_runner.cc:428] Detected that this master 08b7010c899c498e9e6af4db7d5e95d2 is joining an existing cluster
I20260926 12:33:51.081459 17203 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.98.61:38279
I20260926 12:33:51.091349 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.083500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53672 (local address 127.10.98.62:40045)
0926 12:33:51.083628 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.083632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.084177 (+ 545us) server_negotiation.cc:407] Connection header received
0926 12:33:51.084278 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.084281 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.084329 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.084397 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.084846 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.085621 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.086519 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.086745 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.086832 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.088029 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.088033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.088035 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.088063 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.089648 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.090146 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.090150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.090152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.090243 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.090551 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.090555 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.090557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.090747 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.090833 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.091089 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.091202 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 12:33:51.091361 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.083402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.0.0.1:53672)
0926 12:33:51.084095 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.084105 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.084199 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.084418 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.084420 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.084626 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:51.084729 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.084734 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.085741 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.085744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.086407 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.086415 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.086841 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.086844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.086924 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.087441 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:51.087452 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:51.087871 (+ 419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:51.089767 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.089770 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.089772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.090013 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.090350 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.090353 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.090354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.090433 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.090835 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:51.090838 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:51.090933 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.091023 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.091168 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":95,"threads_started":1}
I20260926 12:33:51.092109 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.083828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:38279 (local address 127.0.0.1:49828)
0926 12:33:51.084233 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.084247 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.084349 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.084664 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.084666 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.084881 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:51.085001 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.085008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.085916 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.085920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.086822 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.086832 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.087541 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.087545 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.087634 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.088081 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:51.088096 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:51.088505 (+ 409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:51.090456 (+ 1951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.090459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.090461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.090746 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.091011 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.091014 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.091016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.091082 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.091537 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:51.091540 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:51.091670 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.091770 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.091922 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":78,"threads_started":1}
I20260926 12:33:51.092141 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.083883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49828 (local address 127.10.98.61:38279)
0926 12:33:51.083980 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.083985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.084306 (+ 321us) server_negotiation.cc:407] Connection header received
0926 12:33:51.084458 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.084461 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.084498 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.084543 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.085130 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.085808 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.086945 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.087283 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.087354 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.088646 (+ 1292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.088650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.088652 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.088699 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.090354 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.090848 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.090851 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.090853 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.090925 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.091212 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.091215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.091217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.091401 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.091478 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.091869 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.092015 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 26 12:33:51 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426030, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.62@KRBTEST.COM for oryx/127.10.98.61@KRBTEST.COM
I20260926 12:33:51.106144 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.096252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:38279 (local address 127.0.0.1:49836)
0926 12:33:51.096648 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.096706 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.096839 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.097135 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.097142 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.097594 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:51.097722 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.097729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.098419 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.098423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.099484 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.099496 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.099898 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.099902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.100016 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.100692 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:51.100707 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:51.102379 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:51.104162 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.104172 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.104179 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.104486 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.104761 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.104764 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.104766 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.104855 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.105466 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:51.105473 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:51.105594 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.105743 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.105965 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":109,"threads_started":1}
I20260926 12:33:51.106199 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.096316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49836 (local address 127.10.98.61:38279)
0926 12:33:51.096428 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.096432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.096787 (+ 355us) server_negotiation.cc:407] Connection header received
0926 12:33:51.096957 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.096959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.097002 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.097063 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.097841 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.098315 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.099600 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.099801 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.099878 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.102549 (+ 2671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.102552 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.102554 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.102579 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.104057 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.104596 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.104599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.104600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.104678 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.105060 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.105064 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.105066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.105320 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.105417 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.105903 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.106044 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:33:51.108026 16739 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.98.61:38279: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2"
member_type: NON_VOTER
last_known_addr {
host: "127.10.98.61"
port: 38279
}
attrs {
promote: true
}
}
dest_uuid: "e1e1ab3339074143aa34a2cbe581dc5b"
cas_config_opid_index: -1
I20260926 12:33:51.108789 16739 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: NON_VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: true } }
I20260926 12:33:51.110819 17298 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 08b7010c899c498e9e6af4db7d5e95d2 (127.10.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: NON_VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: true } } }
I20260926 12:33:51.111846 17245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:51.112947 17298 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.98.61:38279
I20260926 12:33:51.113220 17245 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 1 FOLLOWER]: Refusing update from remote peer e1e1ab3339074143aa34a2cbe581dc5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 12:33:51.113330 17299 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: NON_VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: true } } }
I20260926 12:33:51.113432 17299 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:51.113508 17203 master.cc:537] Master@127.10.98.61:38279 shutting down...
I20260926 12:33:51.113926 17299 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: NON_VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
WARNING: no policy specified for oryx/127.10.98.60@KRBTEST.COM; defaulting to no policy
W20260926 12:33:51.116009 16724 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 12:33:51.116900 16724 consensus_peers.cc:487] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b -> Peer 08b7010c899c498e9e6af4db7d5e95d2 (127.10.98.61:38279): Couldn't send request to peer 08b7010c899c498e9e6af4db7d5e95d2. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.98.60@KRBTEST.COM" created.
I20260926 12:33:51.118374 17203 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:51.118678 17203 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:51.118781 17203 tablet_replica.cc:330] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: stopping tablet replica
I20260926 12:33:51.123965 17203 master.cc:552] Master@127.10.98.61:38279 shutdown complete.
I20260926 12:33:51.124022 17203 master_runner.cc:305] Clearing existing system tablet
I20260926 12:33:51.124310 17203 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:51.124378 17203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:51.125918 17203 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:51.126858 17304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:51.127130 17203 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:51.127251 17203 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:51.127290 17203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/wal
uuid: "08b7010c899c498e9e6af4db7d5e95d2"
format_stamp: "Formatted at 2026-09-26 12:33:51 on dist-test-slave-t7tk"
server_key: "02cd0ea6018e414f6c51dba83f8a14e3"
server_key_iv: "8d6406bd9b23dc68133c9bf3266602bd"
server_key_version: "encryptionkey@0"
I20260926 12:33:51.129647 17203 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:51.132541 17203 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 12:33:51.132625 17203 log.cc:1196] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 12:33:51.132877 17203 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: Deleting consensus metadata
I20260926 12:33:51.133049 17203 master_runner.cc:315] Copying system tablet from 127.10.98.62:40045
I20260926 12:33:51.134186 17203 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: tablet copy: Beginning tablet copy session from remote peer at address 127.10.98.62:40045
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.60.keytab.
Entry for principal oryx/127.10.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.60.keytab.
I20260926 12:33:51.137836 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.60:38335
--webserver_interface=127.10.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.60.keytab
--principal=oryx/127.10.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:40045,127.10.98.61:38279,127.10.98.60:38335
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:33:51.143440 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.134880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53680 (local address 127.10.98.62:40045)
0926 12:33:51.135018 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.135022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.135245 (+ 223us) server_negotiation.cc:407] Connection header received
0926 12:33:51.135453 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.135456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.135545 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.135611 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.136532 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.137344 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.138332 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.138590 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.138683 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.139903 (+ 1220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.139908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.139911 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.139940 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.141584 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.142083 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.142087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.142089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.142180 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.142495 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.142498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.142500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.142726 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.142823 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.143153 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.143280 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 12:33:51.143451 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.134764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.0.0.1:53680)
0926 12:33:51.135094 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.135151 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.135327 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.135651 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.135654 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.135938 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:51.136349 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.136361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.137482 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.137486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.138199 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.138212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.138703 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.138707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.138805 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.139286 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:51.139301 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:51.139745 (+ 444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:51.141706 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.141709 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.141711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.141969 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.142297 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.142300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.142302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.142398 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.142821 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:51.142824 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:51.142939 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.143061 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.143233 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":105,"threads_started":1}
I20260926 12:33:51.144011 16759 tablet_copy_service.cc:140] P e1e1ab3339074143aa34a2cbe581dc5b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 08b7010c899c498e9e6af4db7d5e95d2 ({username='oryx', principal='oryx/127.10.98.61@KRBTEST.COM'} at 127.0.0.1:53680)
I20260926 12:33:51.144202 16759 tablet_copy_service.cc:161] P e1e1ab3339074143aa34a2cbe581dc5b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 08b7010c899c498e9e6af4db7d5e95d2 at {username='oryx', principal='oryx/127.10.98.61@KRBTEST.COM'} at 127.0.0.1:53680: session id = 08b7010c899c498e9e6af4db7d5e95d2-00000000000000000000000000000000
I20260926 12:33:51.146134 16759 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 12:33:51.147957 17203 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 12:33:51.150455 17203 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: tablet copy: Starting download of 0 data blocks...
I20260926 12:33:51.150666 17203 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: tablet copy: Starting download of 1 WAL segments...
I20260926 12:33:51.152697 17203 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 12:33:51.154851 16759 tablet_copy_service.cc:342] P e1e1ab3339074143aa34a2cbe581dc5b: Request end of tablet copy session 08b7010c899c498e9e6af4db7d5e95d2-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.98.61@KRBTEST.COM'} at 127.0.0.1:53680
I20260926 12:33:51.155009 16759 tablet_copy_service.cc:433] P e1e1ab3339074143aa34a2cbe581dc5b: ending tablet copy session 08b7010c899c498e9e6af4db7d5e95d2-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 08b7010c899c498e9e6af4db7d5e95d2
I20260926 12:33:51.163278 17203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:51.165364 17315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:51.165555 17312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:51.165683 17313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:51.169250 17203 server_base.cc:1034] running on GCE node
I20260926 12:33:51.169426 17203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:51.169741 17203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:51.170879 17203 hybrid_clock.cc:648] HybridClock initialized: now 1790426031170858 us; error 41 us; skew 500 ppm
Sep 26 12:33:51 dist-test-slave-t7tk krb5kdc[16694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426031, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:51.172569 17203 init.cc:377] Logged in from keytab as oryx/127.10.98.61@KRBTEST.COM (short username oryx)
I20260926 12:33:51.173151 17203 webserver.cc:460] Webserver started at http://127.10.98.61:42735/ using document root <none> and password file <none>
I20260926 12:33:51.173344 17203 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:51.173386 17203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:51.174702 17203 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:51.175683 17322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:51.175928 17203 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:51.176015 17203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/wal
uuid: "08b7010c899c498e9e6af4db7d5e95d2"
format_stamp: "Formatted at 2026-09-26 12:33:51 on dist-test-slave-t7tk"
server_key: "02cd0ea6018e414f6c51dba83f8a14e3"
server_key_iv: "8d6406bd9b23dc68133c9bf3266602bd"
server_key_version: "encryptionkey@0"
I20260926 12:33:51.176098 17203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:51.218070 17203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:51.220306 17203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:51.225198 17203 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.61:38279
I20260926 12:33:51.225811 17203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-1/data/info.pb
I20260926 12:33:51.226222 17374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.61:38279 every 8 connection(s)
I20260926 12:33:51.227689 17375 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:33:51.228763 17375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: Bootstrap starting.
I20260926 12:33:51.239243 17375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:33:51.239527 17375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: Bootstrap complete.
I20260926 12:33:51.240162 17375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: NON_VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: true } }
I20260926 12:33:51.240305 17375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 08b7010c899c498e9e6af4db7d5e95d2, State: Initialized, Role: LEARNER
I20260926 12:33:51.240401 17375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: NON_VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: true } }
I20260926 12:33:51.240741 17378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: NON_VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: true } } }
I20260926 12:33:51.240849 17375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:51.240945 17378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [sys.catalog]: This master's current role is: LEARNER
I20260926 12:33:51.244391 17389 catalog_manager.cc:1259] Loaded cluster ID: f47659f98ed746639f42fd35d409569e
I20260926 12:33:51.244455 17389 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: loading cluster ID for follower catalog manager: success
I20260926 12:33:51.246665 17389 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: acquiring CA information for follower catalog manager: success
I20260926 12:33:51.248031 17389 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 12:33:51.271581 17311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:51.271876 17311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:51.271916 17311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:51.275945 17311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:51.276073 17311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:51.276100 17311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:51.276122 17311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:51.276144 17311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:33:51.281754 17311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:40045,127.10.98.61:38279,127.10.98.60:38335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.60.keytab
--principal=oryx/127.10.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.60:38335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:51.282841 17311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:51.283805 17311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:51.292493 17397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:51.292478 17395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:51.292508 17311 server_base.cc:1034] running on GCE node
W20260926 12:33:51.292464 17394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:51.293398 17311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:51.294533 17311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:51.295715 17311 hybrid_clock.cc:648] HybridClock initialized: now 1790426031295696 us; error 35 us; skew 500 ppm
Sep 26 12:33:51 dist-test-slave-t7tk krb5kdc[16694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426031, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:51.298564 17311 init.cc:377] Logged in from keytab as oryx/127.10.98.60@KRBTEST.COM (short username oryx)
I20260926 12:33:51.299471 17311 webserver.cc:460] Webserver started at http://127.10.98.60:41063/ using document root <none> and password file <none>
I20260926 12:33:51.300005 17311 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:51.300071 17311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:51.300256 17311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:51.302745 17311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/data/instance:
uuid: "fd967af90a354d40bcf567af49879f6f"
format_stamp: "Formatted at 2026-09-26 12:33:51 on dist-test-slave-t7tk"
server_key: "7e4836f61e74d209c55715e7ad8537c6"
server_key_iv: "eacbeefd2164069494bb94aa4732d927"
server_key_version: "encryptionkey@0"
I20260926 12:33:51.303412 17311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/wal/instance:
uuid: "fd967af90a354d40bcf567af49879f6f"
format_stamp: "Formatted at 2026-09-26 12:33:51 on dist-test-slave-t7tk"
server_key: "7e4836f61e74d209c55715e7ad8537c6"
server_key_iv: "eacbeefd2164069494bb94aa4732d927"
server_key_version: "encryptionkey@0"
I20260926 12:33:51.307421 17311 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:33:51.309880 17404 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:51.311287 17311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:33:51.311416 17311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/wal
uuid: "fd967af90a354d40bcf567af49879f6f"
format_stamp: "Formatted at 2026-09-26 12:33:51 on dist-test-slave-t7tk"
server_key: "7e4836f61e74d209c55715e7ad8537c6"
server_key_iv: "eacbeefd2164069494bb94aa4732d927"
server_key_version: "encryptionkey@0"
I20260926 12:33:51.311528 17311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:51.347623 17311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:51.351046 17311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:51.351235 17311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:51.358177 17311 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.60:38335
I20260926 12:33:51.358438 17456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.60:38335 every 8 connection(s)
I20260926 12:33:51.359191 17311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/data/info.pb
I20260926 12:33:51.362201 17457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:51.365475 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 17311
I20260926 12:33:51.366293 17457 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:33:51 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426031, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.60@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM
Sep 26 12:33:51 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.98.60@KRBTEST.COM
I20260926 12:33:51.379660 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.368195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53686 (local address 127.10.98.62:40045)
0926 12:33:51.368306 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.368310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.368742 (+ 432us) server_negotiation.cc:407] Connection header received
0926 12:33:51.369001 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.369004 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.369052 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.369155 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.370106 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.370896 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.372302 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.372654 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.372755 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.375738 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.375741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.375743 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.375769 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.377209 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.377832 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.377835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.377837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.377934 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.378425 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.378428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.378429 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.378678 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.378750 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.379363 (+ 613us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.379493 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:33:51.380676 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.367018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35828 (local address 127.10.98.60:38335)
0926 12:33:51.367572 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.367590 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.367629 (+ 39us) server_negotiation.cc:407] Connection header received
0926 12:33:51.368461 (+ 832us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.368486 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.368837 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.369040 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.369922 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.371181 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.372279 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.372851 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.373088 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.376015 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.376022 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.376035 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.376061 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.377776 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.378239 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.378245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.378247 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.378325 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.378619 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.378621 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.378622 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.379073 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.379249 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.379445 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.379651 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":168,"server-negotiator.queue_time_us":389,"thread_start_us":138,"threads_started":1}
I20260926 12:33:51.380676 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.368052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.0.0.1:53686)
0926 12:33:51.368574 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.368632 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.368827 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.369216 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.369221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.369633 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:51.369973 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.369986 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.371069 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.371071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.372196 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.372205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.372814 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.372817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.373116 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.373763 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:51.373783 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:51.375529 (+ 1746us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:51.377314 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.377321 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.377332 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.377736 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.378135 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.378137 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.378139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.378325 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.378765 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:51.378771 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:51.379129 (+ 358us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.379300 (+ 171us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.379830 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":237,"spinlock_wait_cycles":1792,"thread_start_us":73,"threads_started":1}
I20260926 12:33:51.382651 17457 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:33:51 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426031, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.60@KRBTEST.COM for oryx/127.10.98.61@KRBTEST.COM
I20260926 12:33:51.388720 16930 heartbeater.cc:503] Master 127.10.98.62:40045 was elected leader, sending a full tablet report...
I20260926 12:33:51.391199 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.383083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:38279 (local address 127.0.0.1:49840)
0926 12:33:51.383191 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.383204 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.383268 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.383644 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.383646 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.383764 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:51.383850 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.383854 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.384707 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.384710 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.385318 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.385326 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.385781 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.385784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.385860 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.386321 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:51.386334 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:51.387552 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:51.389484 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.389487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.389489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.389704 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.390143 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.390146 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.390148 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.390226 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.390617 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:51.390619 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:51.390702 (+ 83us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.390827 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.391041 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 12:33:51.391228 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.383192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49840 (local address 127.10.98.61:38279)
0926 12:33:51.383458 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.383461 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.383473 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:33:51.383510 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.383512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.383551 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.383628 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.383932 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.384619 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.385438 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.385669 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.385749 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.387704 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.387708 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.387710 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.387734 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.389363 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.389821 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.389824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.389825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.390009 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.390312 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.390316 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.390318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.390510 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.390591 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.390909 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.391037 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":93,"threads_started":1}
I20260926 12:33:51.392016 17457 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:33:51 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426031, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.60@KRBTEST.COM for oryx/127.10.98.60@KRBTEST.COM
I20260926 12:33:51.399360 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.392467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:38335 (local address 127.0.0.1:35842)
0926 12:33:51.392553 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.392566 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.392632 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.392813 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.392815 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.392980 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:51.393054 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.393058 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.393815 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.393818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.394495 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.394504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.394924 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.394927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.394980 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.395443 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:51.395454 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:51.396567 (+ 1113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:51.398050 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.398053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.398054 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.398221 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.398520 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.398522 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.398523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.398592 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.398940 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:51.398942 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:51.399018 (+ 76us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.399101 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.399196 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 12:33:51.399370 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.392572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35842 (local address 127.10.98.60:38335)
0926 12:33:51.392676 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.392679 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.392690 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:33:51.392715 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.392717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.392747 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.392806 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.393193 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.393698 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.394612 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.394841 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.394918 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.396692 (+ 1774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.396695 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.396697 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.396717 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.397955 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.398361 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.398364 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.398365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.398442 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.398681 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.398683 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.398684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.398848 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.398921 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.399156 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.399242 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:33:51.402694 17457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f: Bootstrap starting.
I20260926 12:33:51.404958 17457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:51.405711 17457 log.cc:824] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:51.407609 17457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f: No bootstrap required, opened a new log
I20260926 12:33:51.410739 17457 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } }
I20260926 12:33:51.410948 17457 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:51.411018 17457 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd967af90a354d40bcf567af49879f6f, State: Initialized, Role: FOLLOWER
I20260926 12:33:51.411439 17457 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } }
I20260926 12:33:51.411870 17463 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } } }
I20260926 12:33:51.412045 17463 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:33:51.412636 17457 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [sys.catalog]: configured and running, proceeding with master startup.
W20260926 12:33:51.417364 17474 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:33:51.417438 17474 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:33:51.421350 17311 master_runner.cc:428] Detected that this master fd967af90a354d40bcf567af49879f6f is joining an existing cluster
I20260926 12:33:51.421411 17311 master_runner.cc:432] Initiating AddMaster RPC to add 127.10.98.60:38335
I20260926 12:33:51.434227 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 16801
I20260926 12:33:51.439640 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:45527
--local_ip_for_outbound_sockets=127.10.98.1
--tserver_master_addrs=127.10.98.62:40045
--webserver_port=46523
--webserver_interface=127.10.98.1
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.1.keytab
--principal=oryx/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.98.62:40045,127.10.98.61:38279
--tserver_master_addrs=127.10.98.62:40045,127.10.98.61:38279,127.10.98.60:38335 with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:33:51.445881 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.423507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.0.0.1:53692)
0926 12:33:51.424000 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.424014 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.424105 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.424595 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.424598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.424879 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:51.425072 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.425080 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.426172 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.426174 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.427442 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.427450 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.428081 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.428084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.428229 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.441651 (+ 13422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:51.441671 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:51.442174 (+ 503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:51.444034 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.444038 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.444040 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.444212 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.444592 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.444595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.444597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.444689 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.445149 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:51.445151 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:51.445265 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.445369 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.445674 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":44,"thread_start_us":101,"threads_started":1}
I20260926 12:33:51.446630 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.424195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53692 (local address 127.10.98.62:40045)
0926 12:33:51.424301 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.424306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.424318 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:33:51.424358 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.424360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.424406 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.424477 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.425219 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.426039 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.427558 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.427893 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.427956 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.442424 (+ 14468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.442429 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.442432 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.442478 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.443911 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.444352 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.444356 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.444358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.444455 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.444785 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.444788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.444790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.445018 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.446224 (+ 1206us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.446311 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.446438 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 12:33:51.448828 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.424401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49848 (local address 127.10.98.61:38279)
0926 12:33:51.424513 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.424517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.424709 (+ 192us) server_negotiation.cc:407] Connection header received
0926 12:33:51.424866 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.424869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.424920 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.424971 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.425554 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.426277 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.427507 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.427839 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.427908 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.443102 (+ 15194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.443106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.443109 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.443142 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.445382 (+ 2240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.446517 (+ 1135us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.446521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.446523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.446610 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.447764 (+ 1154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.447767 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.447769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.447974 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.448040 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.448527 (+ 487us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.448675 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:33:51.449604 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.423603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35856 (local address 127.10.98.60:38335)
0926 12:33:51.423708 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.423712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.424298 (+ 586us) server_negotiation.cc:407] Connection header received
0926 12:33:51.424453 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.424456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.424495 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.424539 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.425198 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.426151 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.427281 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.427652 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.427735 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.442164 (+ 14429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.442168 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.442171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.442215 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.444085 (+ 1870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.447117 (+ 3032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.447120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.447122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.447187 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.447415 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.447418 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.447419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.447629 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.447705 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.449333 (+ 1628us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.449452 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 12:33:51.450269 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.423944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:38279 (local address 127.0.0.1:49848)
0926 12:33:51.424636 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.424648 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.424764 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.425121 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.425124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.425297 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:51.425422 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.425428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.426415 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.426419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.427374 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.427384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.428060 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.428063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.428212 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.428874 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:51.428888 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:51.442622 (+ 13734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:51.445622 (+ 3000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.445625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.445627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.446385 (+ 758us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.446930 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.446933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.446935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.447021 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.448295 (+ 1274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:51.448297 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:51.448359 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.449943 (+ 1584us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.450098 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":37,"thread_start_us":89,"threads_started":1}
I20260926 12:33:51.450882 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.423657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:38335 (local address 127.0.0.1:35856)
0926 12:33:51.424219 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.424233 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.424354 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.424660 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.424663 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.424898 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:51.425037 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.425045 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.426270 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.426273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.427164 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.427174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.427896 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.427900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.428106 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.428582 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:51.428597 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:51.441984 (+ 13387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:51.444625 (+ 2641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.444629 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.444631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.445853 (+ 1222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.447270 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.447272 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.447274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.447341 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.448093 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:51.448096 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:51.448178 (+ 82us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.450596 (+ 2418us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.450724 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":117,"threads_started":1}
Sep 26 12:33:51 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426030, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.62@KRBTEST.COM for oryx/127.10.98.60@KRBTEST.COM
I20260926 12:33:51.461170 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.450914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:38335 (local address 127.0.0.1:35870)
0926 12:33:51.451162 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.451177 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.451257 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.452522 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.452525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.452720 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:51.452825 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.452831 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.453863 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.453867 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.454748 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.454760 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.455319 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.455323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.455422 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.455933 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:51.455948 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:51.457559 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:51.459404 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.459408 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.459410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.459682 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.460010 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.460013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.460015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.460112 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.460524 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:51.460526 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:51.460641 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.460741 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.460893 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169}
I20260926 12:33:51.461184 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.451398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35870 (local address 127.10.98.60:38335)
0926 12:33:51.452192 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.452196 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.452207 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:33:51.452268 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.452271 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.452321 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.452389 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.452949 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.453735 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.454868 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.455210 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.455306 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.457700 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.457704 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.457707 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.457733 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.459280 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.459779 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.459782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.459784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.459867 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.460205 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.460208 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.460210 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.460429 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.460531 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.460798 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.460917 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":728}
I20260926 12:33:51.462558 16739 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.10.98.60:38335: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fd967af90a354d40bcf567af49879f6f"
member_type: NON_VOTER
last_known_addr {
host: "127.10.98.60"
port: 38335
}
attrs {
promote: true
}
}
dest_uuid: "e1e1ab3339074143aa34a2cbe581dc5b"
cas_config_opid_index: 5
I20260926 12:33:51.463227 16739 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: NON_VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: true } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: NON_VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: true } }
I20260926 12:33:51.464310 17299 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fd967af90a354d40bcf567af49879f6f (127.10.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: NON_VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: true } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: NON_VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: true } } }
I20260926 12:33:51.465878 17431 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:51.466019 17299 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.10.98.60:38335
I20260926 12:33:51.466187 17299 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: NON_VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: true } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: NON_VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: true } } }
I20260926 12:33:51.466274 17299 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:51.466930 17311 master.cc:537] Master@127.10.98.60:38335 shutting down...
I20260926 12:33:51.467270 17431 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 1 FOLLOWER]: Refusing update from remote peer e1e1ab3339074143aa34a2cbe581dc5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 12:33:51.468544 17298 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: NON_VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 12:33:51.470564 16726 consensus_peers.cc:487] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b -> Peer fd967af90a354d40bcf567af49879f6f (127.10.98.60:38335): Couldn't send request to peer fd967af90a354d40bcf567af49879f6f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 12:33:51.473228 17311 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:51.473605 17311 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:51.473785 17311 tablet_replica.cc:330] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f: stopping tablet replica
I20260926 12:33:51.477216 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.464717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49864 (local address 127.10.98.61:38279)
0926 12:33:51.464922 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.464927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.469699 (+ 4772us) server_negotiation.cc:407] Connection header received
0926 12:33:51.472494 (+ 2795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.472498 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.472563 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.472626 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:51.473056 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.473863 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.475639 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.476245 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.476342 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.476511 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.476976 (+ 465us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.477064 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260926 12:33:51.477629 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.464466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:38279 (local address 127.0.0.1:49864)
0926 12:33:51.469619 (+ 5153us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.469633 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.472327 (+ 2694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.472813 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.472816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.472835 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:51.472929 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.472935 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.474009 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.474012 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.475432 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.475445 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.476610 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.476613 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.476800 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.476857 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.477412 (+ 555us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.477515 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5059}
I20260926 12:33:51.478240 17349 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 1 LEARNER]: Refusing update from remote peer e1e1ab3339074143aa34a2cbe581dc5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 12:33:51.478706 17298 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: NON_VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 12:33:51.480100 17311 master.cc:552] Master@127.10.98.60:38335 shutdown complete.
I20260926 12:33:51.480147 17311 master_runner.cc:305] Clearing existing system tablet
I20260926 12:33:51.480365 17311 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:51.480408 17311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:51.480710 17349 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER fd967af90a354d40bcf567af49879f6f (127.10.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: NON_VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: true } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: NON_VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: true } } }
I20260926 12:33:51.481652 17311 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 12:33:51.482399 17486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:51.482547 17378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1e1ab3339074143aa34a2cbe581dc5b. Latest consensus state: current_term: 1 leader_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: NON_VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: true } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: NON_VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: true } } }
I20260926 12:33:51.482586 17311 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:51.482651 17378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [sys.catalog]: This master's current role is: LEARNER
I20260926 12:33:51.482690 17311 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:51.482720 17378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: NON_VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: true } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: NON_VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: true } } }
I20260926 12:33:51.482723 17311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/wal
uuid: "fd967af90a354d40bcf567af49879f6f"
format_stamp: "Formatted at 2026-09-26 12:33:51 on dist-test-slave-t7tk"
server_key: "7e4836f61e74d209c55715e7ad8537c6"
server_key_iv: "eacbeefd2164069494bb94aa4732d927"
server_key_version: "encryptionkey@0"
I20260926 12:33:51.482782 17378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [sys.catalog]: This master's current role is: LEARNER
I20260926 12:33:51.484361 17298 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b: attempting to promote NON_VOTER 08b7010c899c498e9e6af4db7d5e95d2 to VOTER
I20260926 12:33:51.484784 17311 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:51.484841 17298 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: NON_VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: true } }
I20260926 12:33:51.486663 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.485961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:38335 (local address 127.0.0.1:35880)
0926 12:33:51.486493 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.486577 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:38335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436}
I20260926 12:33:51.487246 17349 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 1 LEARNER]: Refusing update from remote peer e1e1ab3339074143aa34a2cbe581dc5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 12:33:51.487262 17311 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 12:33:51.487336 17311 log.cc:1196] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 12:33:51.487562 17311 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f: Deleting consensus metadata
I20260926 12:33:51.487834 17311 master_runner.cc:315] Copying system tablet from 127.10.98.62:40045
I20260926 12:33:51.488094 17299 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 12:33:51.488895 17311 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f: tablet copy: Beginning tablet copy session from remote peer at address 127.10.98.62:40045
I20260926 12:33:51.489049 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.488802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:38335 (local address 127.0.0.1:35892)
0926 12:33:51.488932 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.488979 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:38335: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 12:33:51.489805 16726 consensus_peers.cc:487] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b -> Peer fd967af90a354d40bcf567af49879f6f (127.10.98.60:38335): Couldn't send request to peer fd967af90a354d40bcf567af49879f6f. Status: Network error: Client connection negotiation failed: client connection to 127.10.98.60:38335: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 12:33:51.490800 17299 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 08b7010c899c498e9e6af4db7d5e95d2 (127.10.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: NON_VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: true } } }
I20260926 12:33:51.492213 17299 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: NON_VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: true } } }
I20260926 12:33:51.492309 17299 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:51.492435 17299 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: NON_VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: true } } }
I20260926 12:33:51.492512 17299 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:51.493419 17349 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 08b7010c899c498e9e6af4db7d5e95d2 (127.10.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: NON_VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: true } } }
I20260926 12:33:51.494786 17487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: NON_VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: true } } }
I20260926 12:33:51.494913 17487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:33:51.498350 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.489559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53700 (local address 127.10.98.62:40045)
0926 12:33:51.489658 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.489662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.490173 (+ 511us) server_negotiation.cc:407] Connection header received
0926 12:33:51.490341 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.490345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.490407 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.490453 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.491096 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.491896 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.493071 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.493476 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.493548 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.494905 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.494909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.494912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.494939 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.496682 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.497175 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.497178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.497180 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.497267 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.497537 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.497540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.497542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.497744 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.497833 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.498095 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.498203 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 12:33:51.498705 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.489749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.0.0.1:53700)
0926 12:33:51.490095 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.490110 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.490243 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.490627 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.490630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.490848 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:51.490955 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.490962 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.492070 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.492073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.492941 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.492953 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.493700 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.493703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.493780 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.494312 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:51.494326 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:51.494759 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:51.496798 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.496801 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.496804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.497055 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.497353 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.497356 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.497358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.497449 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.497829 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:51.497832 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:51.497936 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.498039 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.498181 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":85,"threads_started":1}
I20260926 12:33:51.499266 16759 tablet_copy_service.cc:140] P e1e1ab3339074143aa34a2cbe581dc5b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fd967af90a354d40bcf567af49879f6f ({username='oryx', principal='oryx/127.10.98.60@KRBTEST.COM'} at 127.0.0.1:53700)
I20260926 12:33:51.499351 16759 tablet_copy_service.cc:161] P e1e1ab3339074143aa34a2cbe581dc5b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fd967af90a354d40bcf567af49879f6f at {username='oryx', principal='oryx/127.10.98.60@KRBTEST.COM'} at 127.0.0.1:53700: session id = fd967af90a354d40bcf567af49879f6f-00000000000000000000000000000000
I20260926 12:33:51.500353 16759 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 12:33:51.501844 17311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 12:33:51.504089 17311 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f: tablet copy: Starting download of 0 data blocks...
I20260926 12:33:51.504253 17311 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f: tablet copy: Starting download of 1 WAL segments...
I20260926 12:33:51.505851 17311 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 12:33:51.507570 16759 tablet_copy_service.cc:342] P e1e1ab3339074143aa34a2cbe581dc5b: Request end of tablet copy session fd967af90a354d40bcf567af49879f6f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.10.98.60@KRBTEST.COM'} at 127.0.0.1:53700
I20260926 12:33:51.507645 16759 tablet_copy_service.cc:433] P e1e1ab3339074143aa34a2cbe581dc5b: ending tablet copy session fd967af90a354d40bcf567af49879f6f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fd967af90a354d40bcf567af49879f6f
I20260926 12:33:51.514410 17311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:51.516396 17496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:51.516486 17495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:51.517123 17498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:51.521198 17311 server_base.cc:1034] running on GCE node
I20260926 12:33:51.521411 17311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:51.521744 17311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:51.522882 17311 hybrid_clock.cc:648] HybridClock initialized: now 1790426031522869 us; error 34 us; skew 500 ppm
Sep 26 12:33:51 dist-test-slave-t7tk krb5kdc[16694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426031, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:51.524425 17311 init.cc:377] Logged in from keytab as oryx/127.10.98.60@KRBTEST.COM (short username oryx)
I20260926 12:33:51.524930 17311 webserver.cc:460] Webserver started at http://127.10.98.60:42841/ using document root <none> and password file <none>
I20260926 12:33:51.525153 17311 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:51.525203 17311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:51.526587 17311 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:51.527702 17505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:51.527959 17311 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:51.528064 17311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/wal
uuid: "fd967af90a354d40bcf567af49879f6f"
format_stamp: "Formatted at 2026-09-26 12:33:51 on dist-test-slave-t7tk"
server_key: "7e4836f61e74d209c55715e7ad8537c6"
server_key_iv: "eacbeefd2164069494bb94aa4732d927"
server_key_version: "encryptionkey@0"
I20260926 12:33:51.528184 17311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:51.541467 17311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:51.543805 17311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:51.548908 17311 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.60:38335
I20260926 12:33:51.548946 17557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.60:38335 every 8 connection(s)
I20260926 12:33:51.549715 17311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/master-2/data/info.pb
I20260926 12:33:51.551740 17558 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:33:51.552809 17558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f: Bootstrap starting.
I20260926 12:33:51.565164 17558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:33:51.565474 17558 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f: Bootstrap complete.
I20260926 12:33:51.566099 17558 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: NON_VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: true } }
I20260926 12:33:51.566256 17558 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fd967af90a354d40bcf567af49879f6f, State: Initialized, Role: LEARNER
I20260926 12:33:51.566362 17558 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: NON_VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: true } }
I20260926 12:33:51.566849 17558 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:51.567164 17561 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: NON_VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: true } } }
I20260926 12:33:51.567365 17561 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [sys.catalog]: This master's current role is: LEARNER
I20260926 12:33:51.570581 17572 catalog_manager.cc:1259] Loaded cluster ID: f47659f98ed746639f42fd35d409569e
I20260926 12:33:51.570643 17572 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f: loading cluster ID for follower catalog manager: success
I20260926 12:33:51.572727 17572 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f: acquiring CA information for follower catalog manager: success
I20260926 12:33:51.574226 17572 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 12:33:51.583987 17483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:51.584275 17483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:51.584335 17483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:51.588514 17483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:51.588585 17483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:51.588668 17483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:33:51.594414 17483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.1.keytab
--principal=oryx/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:45527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=46523
--tserver_master_addrs=127.10.98.62:40045,127.10.98.61:38279,127.10.98.60:38335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:51.595504 17483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:51.596511 17483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:51.606504 17578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:51.606694 17483 server_base.cc:1034] running on GCE node
W20260926 12:33:51.606489 17577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:51.606799 17580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:51.607304 17483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:51.607977 17483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:51.609162 17483 hybrid_clock.cc:648] HybridClock initialized: now 1790426031609134 us; error 45 us; skew 500 ppm
Sep 26 12:33:51 dist-test-slave-t7tk krb5kdc[16694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426031, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:51.611869 17483 init.cc:377] Logged in from keytab as oryx/127.10.98.1@KRBTEST.COM (short username oryx)
I20260926 12:33:51.612795 17483 webserver.cc:460] Webserver started at http://127.10.98.1:46523/ using document root <none> and password file <none>
I20260926 12:33:51.613404 17483 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:51.613488 17483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:51.617584 17483 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:33:51.619669 17587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:51.620960 17483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:33:51.621150 17483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "d4f592a9801e467da0bc143fe5c41cbc"
format_stamp: "Formatted at 2026-09-26 12:33:50 on dist-test-slave-t7tk"
server_key: "8faafd2e8be705fda2c3447c66358528"
server_key_iv: "eb2f7e368507479fc4c3e819747dcfbe"
server_key_version: "encryptionkey@0"
I20260926 12:33:51.621690 17483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:51.628566 17064 heartbeater.cc:503] Master 127.10.98.62:40045 was elected leader, sending a full tablet report...
I20260926 12:33:51.639624 17483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:51.643448 17483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:51.643635 17483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:51.644366 17483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:51.645380 17483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:51.645437 17483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:51.645479 17483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:51.645506 17483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:51.653788 17483 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:45527
I20260926 12:33:51.653831 17700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:45527 every 8 connection(s)
I20260926 12:33:51.654824 17483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:33:51.663075 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 17483
I20260926 12:33:51.663266 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 16935
Sep 26 12:33:51 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426031, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.61@KRBTEST.COM
Sep 26 12:33:51 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426031, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.60@KRBTEST.COM
I20260926 12:33:51.670120 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:33289
--local_ip_for_outbound_sockets=127.10.98.2
--tserver_master_addrs=127.10.98.62:40045
--webserver_port=41627
--webserver_interface=127.10.98.2
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.2.keytab
--principal=oryx/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.98.62:40045,127.10.98.61:38279
--tserver_master_addrs=127.10.98.62:40045,127.10.98.61:38279,127.10.98.60:38335 with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:33:51.670696 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.656900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:34343 (local address 127.10.98.61:38279)
0926 12:33:51.657012 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.657015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.658014 (+ 999us) server_negotiation.cc:407] Connection header received
0926 12:33:51.659187 (+ 1173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.659190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.659242 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.659295 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.660931 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.661591 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.663040 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.663354 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.663446 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.666514 (+ 3068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.666518 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.666520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.666551 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.668186 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.668842 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.668845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.668847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.668937 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.669367 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.669371 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.669373 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.669594 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.669667 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.670405 (+ 738us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.670527 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 12:33:51 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426031, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM
I20260926 12:33:51.672189 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.656680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:44141 (local address 127.10.98.60:38335)
0926 12:33:51.656924 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.656936 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.658015 (+ 1079us) server_negotiation.cc:407] Connection header received
0926 12:33:51.659187 (+ 1172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.659190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.659242 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.659299 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.660883 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.661993 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.663296 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.663627 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.663692 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.667798 (+ 4106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.667802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.667805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.667834 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.669666 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.670143 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.670147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.670149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.670237 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.671165 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.671168 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.671170 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.671395 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.671468 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.671875 (+ 407us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.672024 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
I20260926 12:33:51.672427 17706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.657306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:38279 (local address 127.10.98.1:34343)
0926 12:33:51.657845 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.657907 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.658944 (+ 1037us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.659476 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.659483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.660020 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:51.660699 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.660776 (+ 77us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.661729 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.661731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.662895 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.662907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.663931 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.663935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.664136 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.664689 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:51.664706 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:51.666316 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:51.668316 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.668323 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.668340 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.668756 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.669037 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.669040 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.669042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.669259 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.669740 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:51.669747 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:51.670106 (+ 359us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.670879 (+ 773us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.671274 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":162,"spinlock_wait_cycles":11136,"thread_start_us":134,"threads_started":1}
I20260926 12:33:51.673336 17707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.657306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:38335 (local address 127.10.98.1:44141)
0926 12:33:51.657845 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.657907 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.658943 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.659476 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.659483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.660016 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:51.660699 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.660766 (+ 67us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.662069 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.662072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.663069 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.663079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.665049 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.665053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.665174 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.665654 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:51.665668 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:51.667639 (+ 1971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:51.669786 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.669789 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.669791 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.670018 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.670665 (+ 647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.670668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.670670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.671053 (+ 383us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.671593 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:51.671596 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:51.671691 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.672735 (+ 1044us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.672875 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":212,"spinlock_wait_cycles":94080,"thread_start_us":158,"threads_started":1}
I20260926 12:33:51.674258 17703 heartbeater.cc:347] Connected to a master server at 127.10.98.61:38279
I20260926 12:33:51.674487 17701 heartbeater.cc:347] Connected to a master server at 127.10.98.60:38335
I20260926 12:33:51.674489 17703 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:51.674592 17701 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:51.675153 17701 heartbeater.cc:511] Master 127.10.98.60:38335 requested a full tablet report, sending...
I20260926 12:33:51.675153 17703 heartbeater.cc:511] Master 127.10.98.61:38279 requested a full tablet report, sending...
I20260926 12:33:51.676926 17514 ts_manager.cc:194] Registered new tserver with Master: d4f592a9801e467da0bc143fe5c41cbc (127.10.98.1:45527)
I20260926 12:33:51.677685 17339 ts_manager.cc:194] Registered new tserver with Master: d4f592a9801e467da0bc143fe5c41cbc (127.10.98.1:45527)
I20260926 12:33:51.678807 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.658536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:51337 (local address 127.10.98.62:40045)
0926 12:33:51.658659 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.658664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.658734 (+ 70us) server_negotiation.cc:407] Connection header received
0926 12:33:51.659162 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.659166 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.659222 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.659293 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.660851 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.661758 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.663208 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.663537 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.663610 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.671807 (+ 8197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.671811 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.671814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.671845 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.673988 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.674945 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.674949 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.674951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.675047 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.675419 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.675423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.675425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.675654 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.675726 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.678590 (+ 2864us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.678697 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:33:51.678833 17708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.658406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.1:51337)
0926 12:33:51.658667 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.658679 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.659004 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.659475 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.659482 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.660016 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:51.660696 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.660714 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.661862 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.661865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.663069 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.663079 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.669709 (+ 6630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.669712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.669817 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.670233 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:51.670245 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:51.671637 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:51.674562 (+ 2925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.674565 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.674567 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.674824 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.675161 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.675164 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.675166 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.675301 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.675934 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:51.675937 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:51.676027 (+ 90us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.678504 (+ 2477us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.678656 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":159,"thread_start_us":81,"threads_started":1}
I20260926 12:33:51.679669 17702 heartbeater.cc:347] Connected to a master server at 127.10.98.62:40045
I20260926 12:33:51.679754 17702 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:51.679947 17702 heartbeater.cc:511] Master 127.10.98.62:40045 requested a full tablet report, sending...
I20260926 12:33:51.680752 16739 ts_manager.cc:194] Re-registered known tserver with Master: d4f592a9801e467da0bc143fe5c41cbc (127.10.98.1:45527)
I20260926 12:33:51.681358 16739 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:51337
W20260926 12:33:51.782968 17709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:51.783257 17709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:51.783318 17709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:51.787453 17709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:51.787521 17709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:51.787600 17709 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:33:51.793138 17709 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.2.keytab
--principal=oryx/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:33289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=41627
--tserver_master_addrs=127.10.98.62:40045,127.10.98.61:38279,127.10.98.60:38335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17709
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:51.794224 17709 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:51.795257 17709 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:33:51.804845 17709 server_base.cc:1034] running on GCE node
W20260926 12:33:51.804778 17714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:51.804756 17715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:51.804781 17717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:51.805503 17709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:51.806138 17709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:51.807328 17709 hybrid_clock.cc:648] HybridClock initialized: now 1790426031807288 us; error 53 us; skew 500 ppm
Sep 26 12:33:51 dist-test-slave-t7tk krb5kdc[16694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426031, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:51.810238 17709 init.cc:377] Logged in from keytab as oryx/127.10.98.2@KRBTEST.COM (short username oryx)
I20260926 12:33:51.811149 17709 webserver.cc:460] Webserver started at http://127.10.98.2:41627/ using document root <none> and password file <none>
I20260926 12:33:51.811689 17709 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:51.811769 17709 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:51.815768 17709 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:33:51.817893 17724 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:51.819260 17709 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:33:51.819365 17709 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "08fc683fa8894fa685caba488359483d"
format_stamp: "Formatted at 2026-09-26 12:33:50 on dist-test-slave-t7tk"
server_key: "d9c4eed500a21070027c5583819e04d2"
server_key_iv: "49e5bf16dd10954f932b8d54d0d5912b"
server_key_version: "encryptionkey@0"
I20260926 12:33:51.820019 17709 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:51.832259 17709 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:51.832499 17198 heartbeater.cc:503] Master 127.10.98.62:40045 was elected leader, sending a full tablet report...
I20260926 12:33:51.835690 17709 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:51.835878 17709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:51.836480 17709 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:51.837404 17709 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:51.837450 17709 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:51.837507 17709 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:51.837530 17709 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:51.846534 17709 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:33289
I20260926 12:33:51.846549 17837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:33289 every 8 connection(s)
I20260926 12:33:51.847529 17709 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:33:51.857414 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 17709
I20260926 12:33:51.857606 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 17069
Sep 26 12:33:51 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426031, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.60@KRBTEST.COM
Sep 26 12:33:51 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426031, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.61@KRBTEST.COM
Sep 26 12:33:51 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426031, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM
I20260926 12:33:51.864758 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.851885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:48379 (local address 127.10.98.61:38279)
0926 12:33:51.852005 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.852010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.852023 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:33:51.852235 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.852238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.852286 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.852349 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.853786 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.854573 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.856356 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.856680 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.856773 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.860669 (+ 3896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.860673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.860676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.860707 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.862744 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.863170 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.863173 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.863175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.863248 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.863515 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.863519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.863520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.863731 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.863791 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.864456 (+ 665us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.864590 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 12:33:51.864758 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.849409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:51575 (local address 127.10.98.60:38335)
0926 12:33:51.849534 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.849537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.850848 (+ 1311us) server_negotiation.cc:407] Connection header received
0926 12:33:51.852064 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.852068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.852125 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.852202 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.854714 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.855217 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.856116 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.856385 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.856480 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.860554 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.860559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.860562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.860616 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.862210 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.862927 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.862931 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.862933 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.863016 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.863420 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.863423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.863425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.863654 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.863723 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.864466 (+ 743us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.864590 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:33:51.865288 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:46769
--local_ip_for_outbound_sockets=127.10.98.3
--tserver_master_addrs=127.10.98.62:40045
--webserver_port=38669
--webserver_interface=127.10.98.3
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.3.keytab
--principal=oryx/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.10.98.62:40045,127.10.98.61:38279
--tserver_master_addrs=127.10.98.62:40045,127.10.98.61:38279,127.10.98.60:38335 with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:33:51.866880 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.851285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:51631 (local address 127.10.98.62:40045)
0926 12:33:51.851422 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.851427 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.851510 (+ 83us) server_negotiation.cc:407] Connection header received
0926 12:33:51.852084 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.852087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.852133 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.852181 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:51.853763 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.854601 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.856284 (+ 1683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.856594 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.856684 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.863623 (+ 6939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:51.863628 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:51.863630 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:51.863662 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:51.865243 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.865671 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.865674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.865676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.865760 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:51.866043 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:51.866047 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:51.866049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:51.866259 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:51.866343 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.866626 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.866741 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 12:33:51.867034 17842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.849805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:38335 (local address 127.10.98.2:51575)
0926 12:33:51.850683 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.850728 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.851828 (+ 1100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.852485 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.852494 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.852948 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:51.853631 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.853640 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.855325 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.855328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.856001 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.856009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.856521 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.856524 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.856715 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.857413 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:51.857434 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:51.860335 (+ 2901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:51.862348 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.862354 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.862372 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.862822 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.863137 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.863140 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.863142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.863322 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.863803 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:51.863811 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:51.864171 (+ 360us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.864918 (+ 747us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.866309 (+ 1391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":1102,"spinlock_wait_cycles":1920,"thread_start_us":524,"threads_started":1}
I20260926 12:33:51.867443 17843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.851170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.2:51631)
0926 12:33:51.851416 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.851431 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.851835 (+ 404us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.852484 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.852492 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.852944 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:51.853623 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.853637 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.854729 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.854732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.856168 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.856179 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.856697 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.856700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.856788 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.861992 (+ 5204us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:51.862007 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:51.863434 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:51.865338 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.865342 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.865344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.865556 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.865865 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.865867 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.865869 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.865946 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.866350 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:51.866353 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:51.866442 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.867091 (+ 649us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.867229 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":203,"thread_start_us":77,"threads_started":1}
I20260926 12:33:51.867962 17844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.851568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:38279 (local address 127.10.98.2:48379)
0926 12:33:51.851824 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.851838 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.852131 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.852485 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.852494 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.852947 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:51.853634 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.853640 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.854729 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.854732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.856231 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.856238 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.856903 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.856906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.857013 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.857596 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:51.857610 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:51.860509 (+ 2899us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:51.862858 (+ 2349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.862861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.862863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.863072 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.863348 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:51.863350 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:51.863352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:51.863424 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:51.864118 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:51.864122 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:51.864208 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.865003 (+ 795us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.865379 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":216,"spinlock_wait_cycles":11776,"thread_start_us":72,"threads_started":1}
I20260926 12:33:51.868326 17838 heartbeater.cc:347] Connected to a master server at 127.10.98.60:38335
I20260926 12:33:51.868571 17838 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:51.868870 17840 heartbeater.cc:347] Connected to a master server at 127.10.98.61:38279
I20260926 12:33:51.868958 17840 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:51.869158 17839 heartbeater.cc:347] Connected to a master server at 127.10.98.62:40045
I20260926 12:33:51.869161 17838 heartbeater.cc:511] Master 127.10.98.60:38335 requested a full tablet report, sending...
I20260926 12:33:51.869163 17840 heartbeater.cc:511] Master 127.10.98.61:38279 requested a full tablet report, sending...
I20260926 12:33:51.869354 17839 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:51.869541 17839 heartbeater.cc:511] Master 127.10.98.62:40045 requested a full tablet report, sending...
I20260926 12:33:51.870306 16739 ts_manager.cc:194] Re-registered known tserver with Master: 08fc683fa8894fa685caba488359483d (127.10.98.2:33289)
I20260926 12:33:51.870369 17514 ts_manager.cc:194] Registered new tserver with Master: 08fc683fa8894fa685caba488359483d (127.10.98.2:33289)
I20260926 12:33:51.870364 17339 ts_manager.cc:194] Registered new tserver with Master: 08fc683fa8894fa685caba488359483d (127.10.98.2:33289)
I20260926 12:33:51.871040 16739 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:51631
I20260926 12:33:51.953454 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.950128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:38335 (local address 127.0.0.1:35894)
0926 12:33:51.950300 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:51.950314 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:51.950400 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:51.950727 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:51.950730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:51.950750 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:51.950844 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.950849 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.951587 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.951591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.952332 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:51.952341 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.953007 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.953011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:51.953133 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.953155 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:33:51.953225 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 12:33:51.953318 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 12:33:51.953454 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:51.950242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35894 (local address 127.10.98.60:38335)
0926 12:33:51.950397 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:51.950403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:51.950421 (+ 18us) server_negotiation.cc:407] Connection header received
0926 12:33:51.950559 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:51.950563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:51.950619 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:51.950695 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:51.950950 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.951479 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.952453 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:51.952880 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:51.952982 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:51.953185 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:51.953245 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 12:33:51.953312 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260926 12:33:51.954157 17532 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 12:33:51.954547 17561 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1e1ab3339074143aa34a2cbe581dc5b. Latest consensus state: current_term: 1 leader_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: NON_VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: true } } }
I20260926 12:33:51.954692 17561 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [sys.catalog]: This master's current role is: LEARNER
W20260926 12:33:51.978951 17845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:51.979209 17845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:51.979251 17845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:51.983333 17845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:51.983399 17845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:51.983471 17845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:33:51.988708 17845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.3.keytab
--principal=oryx/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:46769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=38669
--tserver_master_addrs=127.10.98.62:40045,127.10.98.61:38279,127.10.98.60:38335
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:51.989821 17845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:51.990886 17845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:52.001227 17850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:52.001248 17853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:52.001226 17851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:52.001436 17845 server_base.cc:1034] running on GCE node
I20260926 12:33:52.002015 17845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:52.002696 17845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:52.003916 17845 hybrid_clock.cc:648] HybridClock initialized: now 1790426032003896 us; error 38 us; skew 500 ppm
Sep 26 12:33:52 dist-test-slave-t7tk krb5kdc[16694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426032, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:52.006657 17845 init.cc:377] Logged in from keytab as oryx/127.10.98.3@KRBTEST.COM (short username oryx)
I20260926 12:33:52.007490 17845 webserver.cc:460] Webserver started at http://127.10.98.3:38669/ using document root <none> and password file <none>
I20260926 12:33:52.008015 17845 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:52.008085 17845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:52.011883 17845 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:33:52.014091 17860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:52.015359 17845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:33:52.015496 17845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "46015c6e40794d6486049f51816f2200"
format_stamp: "Formatted at 2026-09-26 12:33:50 on dist-test-slave-t7tk"
server_key: "113cfb34f2dba6d61572d21c278ffe71"
server_key_iv: "8b1c24061c6dda1a402d8aaa0adb5572"
server_key_version: "encryptionkey@0"
I20260926 12:33:52.016036 17845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:52.030189 17845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:52.033470 17845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:52.033682 17845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:52.034255 17845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:33:52.035100 17845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:52.035147 17845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:52.035187 17845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:52.035207 17845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:52.043905 17845 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:46769
I20260926 12:33:52.043910 17973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:46769 every 8 connection(s)
I20260926 12:33:52.045080 17845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:33:52.052573 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 17845
Sep 26 12:33:52 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426032, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM
Sep 26 12:33:52 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426032, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.60@KRBTEST.COM
Sep 26 12:33:52 dist-test-slave-t7tk krb5kdc[16694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426032, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.61@KRBTEST.COM
I20260926 12:33:52.063930 17986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.054510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.10.98.62:40045)
0926 12:33:52.055056 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:52.055060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:52.055082 (+ 22us) server_negotiation.cc:407] Connection header received
0926 12:33:52.055115 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:52.055118 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:52.055164 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:52.055252 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:52.056109 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.056904 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.058095 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.058413 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.058523 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.060202 (+ 1679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:52.060206 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:52.060209 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:52.060239 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:52.061853 (+ 1614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:52.062304 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:52.062307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:52.062309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:52.062405 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:52.062701 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:52.062704 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:52.062706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:52.062921 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:52.063046 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:52.063624 (+ 578us) server_negotiation.cc:299] Negotiation successful
0926 12:33:52.063754 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"spinlock_wait_cycles":4352,"thread_start_us":66,"threads_started":1}
I20260926 12:33:52.065156 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.046902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:36661 (local address 127.10.98.60:38335)
0926 12:33:52.047035 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:52.047039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:52.048249 (+ 1210us) server_negotiation.cc:407] Connection header received
0926 12:33:52.049276 (+ 1027us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:52.049280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:52.049327 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:52.049405 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:52.050762 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.051601 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.052875 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.053213 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.053289 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.058635 (+ 5346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:52.058639 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:52.058642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:52.058669 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:52.062958 (+ 4289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:52.063393 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:52.063396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:52.063398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:52.063477 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:52.063762 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:52.063766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:52.063767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:52.063985 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:52.064064 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:52.064868 (+ 804us) server_negotiation.cc:299] Negotiation successful
0926 12:33:52.064977 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:33:52.065156 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.046949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:52529 (local address 127.10.98.61:38279)
0926 12:33:52.047138 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:52.047142 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:52.048023 (+ 881us) server_negotiation.cc:407] Connection header received
0926 12:33:52.049163 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:52.049166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:52.049224 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:52.049302 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:52.050793 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.051601 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.052906 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.053268 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.053339 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.060874 (+ 7535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:52.060878 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:52.060880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:52.060908 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:52.062736 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:52.063247 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:52.063250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:52.063252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:52.063330 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:52.063785 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:52.063788 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:52.063790 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:52.064004 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:52.064064 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:52.064856 (+ 792us) server_negotiation.cc:299] Negotiation successful
0926 12:33:52.064977 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260926 12:33:52.066012 17978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.047359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:38279 (local address 127.10.98.3:52529)
0926 12:33:52.047845 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:52.047897 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:52.048911 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:52.049493 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:52.049500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:52.050026 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:52.050628 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:52.050637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.051711 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.051714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:52.052793 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:52.052800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.053515 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.053519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:52.053795 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.054666 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:52.054689 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:52.060709 (+ 6020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:52.062851 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:52.062855 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:52.062857 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:52.063140 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:52.063443 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:52.063446 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:52.063448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:52.063692 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:52.064596 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:52.064598 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:52.064678 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 12:33:52.065623 (+ 945us) client_negotiation.cc:240] Negotiation successful
0926 12:33:52.065761 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":236,"spinlock_wait_cycles":18048,"thread_start_us":536,"threads_started":1}
I20260926 12:33:52.066594 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.048508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:55113 (local address 127.10.98.62:40045)
0926 12:33:52.048625 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:52.048629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:52.048717 (+ 88us) server_negotiation.cc:407] Connection header received
0926 12:33:52.049526 (+ 809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:52.049529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:52.049577 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:52.049660 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:52.050784 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.051318 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.052883 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.053265 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.053339 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.058664 (+ 5325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:52.058668 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:52.058671 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:52.058705 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:52.062406 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:52.063260 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:52.063264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:52.063265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:52.063344 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:52.063735 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:52.063738 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:52.063739 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:52.063916 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:52.063988 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:52.066332 (+ 2344us) server_negotiation.cc:299] Negotiation successful
0926 12:33:52.066449 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50,"spinlock_wait_cycles":1792}
I20260926 12:33:52.067056 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.047359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:38335 (local address 127.10.98.3:36661)
0926 12:33:52.048158 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:52.048171 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:52.048952 (+ 781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:52.049494 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:52.049502 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:52.050026 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:52.050621 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:52.050636 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.051748 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.051752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:52.052742 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:52.052753 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.053515 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.053520 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:52.053855 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.055118 (+ 1263us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:52.055132 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:52.058524 (+ 3392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:52.063069 (+ 4545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:52.063072 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:52.063074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:52.063293 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:52.063570 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:52.063573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:52.063574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:52.063663 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:52.064357 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:52.064361 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:52.064457 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 12:33:52.064596 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 12:33:52.066880 (+ 2284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":2299,"spinlock_wait_cycles":36736,"thread_start_us":130,"threads_started":1}
I20260926 12:33:52.067678 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.048377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.3:55113)
0926 12:33:52.048607 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:52.048621 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:52.048916 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:52.049698 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:52.049700 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:52.050026 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:52.050632 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:52.050639 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.051468 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.051470 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:52.052747 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:52.052758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.053515 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.053519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:52.053799 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.054890 (+ 1091us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:52.054902 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:52.058173 (+ 3271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:52.062719 (+ 4546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:52.062725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:52.062745 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:52.063140 (+ 395us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:52.063440 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:52.063443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:52.063445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:52.063635 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:52.064064 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:52.064071 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:52.064381 (+ 310us) client_negotiation.cc:769] Sending connection context
0926 12:33:52.064596 (+ 215us) client_negotiation.cc:240] Negotiation successful
0926 12:33:52.064968 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":178,"spinlock_wait_cycles":49024,"thread_start_us":80,"threads_started":1}
I20260926 12:33:52.068998 17975 heartbeater.cc:347] Connected to a master server at 127.10.98.62:40045
I20260926 12:33:52.069276 17975 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:52.069844 17975 heartbeater.cc:511] Master 127.10.98.62:40045 requested a full tablet report, sending...
I20260926 12:33:52.070993 16739 ts_manager.cc:194] Re-registered known tserver with Master: 46015c6e40794d6486049f51816f2200 (127.10.98.3:46769)
I20260926 12:33:52.071566 16739 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:55113
I20260926 12:33:52.073145 17974 heartbeater.cc:347] Connected to a master server at 127.10.98.60:38335
I20260926 12:33:52.073243 17974 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:52.073493 17974 heartbeater.cc:511] Master 127.10.98.60:38335 requested a full tablet report, sending...
I20260926 12:33:52.073598 17976 heartbeater.cc:347] Connected to a master server at 127.10.98.61:38279
I20260926 12:33:52.073658 17976 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:52.073783 17976 heartbeater.cc:511] Master 127.10.98.61:38279 requested a full tablet report, sending...
I20260926 12:33:52.074324 17514 ts_manager.cc:194] Registered new tserver with Master: 46015c6e40794d6486049f51816f2200 (127.10.98.3:46769)
I20260926 12:33:52.074833 17339 ts_manager.cc:194] Registered new tserver with Master: 46015c6e40794d6486049f51816f2200 (127.10.98.3:46769)
I20260926 12:33:52.076789 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.055107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35906 (local address 127.10.98.60:38335)
0926 12:33:52.055303 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:52.055314 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:52.055530 (+ 216us) server_negotiation.cc:407] Connection header received
0926 12:33:52.055619 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:52.055622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:52.055670 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:52.055743 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:52.056582 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.057347 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.058409 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.058728 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.058812 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.062577 (+ 3765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:52.062581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:52.062584 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:52.062614 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:52.075000 (+ 12386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:52.075506 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:52.075509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:52.075511 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:52.075607 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:52.075845 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:52.075848 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:52.075849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:52.076067 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:52.076214 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:52.076496 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 12:33:52.076626 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":51,"threads_started":1}
I20260926 12:33:52.074008 16739 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53714:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:33:52.078790 16739 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:33:52.081650 17992 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: NON_VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: true } } }
I20260926 12:33:52.081792 17992 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:52.085062 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.054890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49872 (local address 127.10.98.61:38279)
0926 12:33:52.055103 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:52.055107 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:52.055919 (+ 812us) server_negotiation.cc:407] Connection header received
0926 12:33:52.055959 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:52.055962 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:52.056008 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:52.056084 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:52.056994 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.057835 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.059233 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.059561 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.059656 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.061955 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:52.061959 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:52.061961 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:52.061991 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:52.082983 (+ 20992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:52.083543 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:52.083547 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:52.083549 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:52.083647 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:52.083993 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:52.083996 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:52.083997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:52.084247 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:52.084387 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:52.084747 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 12:33:52.084887 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":63,"threads_started":1}
I20260926 12:33:52.087893 17992 raft_consensus.cc:1050] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b: attempting to promote NON_VOTER fd967af90a354d40bcf567af49879f6f to VOTER
I20260926 12:33:52.088263 17992 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false } }
I20260926 12:33:52.089563 17532 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 1 LEARNER]: Refusing update from remote peer e1e1ab3339074143aa34a2cbe581dc5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 12:33:52.089902 17349 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 1 FOLLOWER]: Refusing update from remote peer e1e1ab3339074143aa34a2cbe581dc5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 12:33:52.090198 17993 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 12:33:52.090360 17488 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 12:33:52.092267 17488 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fd967af90a354d40bcf567af49879f6f (127.10.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false } } }
I20260926 12:33:52.092697 17349 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fd967af90a354d40bcf567af49879f6f (127.10.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false } } }
I20260926 12:33:52.093719 17488 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false } } }
I20260926 12:33:52.093815 17488 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:52.093928 17488 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false } } }
I20260926 12:33:52.094000 17488 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:52.094499 17992 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false } } }
I20260926 12:33:52.094609 17992 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:52.094813 17532 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, fd967af90a354d40bcf567af49879f6f (127.10.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false } } }
I20260926 12:33:52.096624 17561 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false } } }
I20260926 12:33:52.096741 17561 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:33:52.100184 18001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false } } }
I20260926 12:33:52.100349 18001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:33:52.102672 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.096171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:45527 (local address 127.0.0.1:54632)
0926 12:33:52.096323 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:52.096339 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:52.096431 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:52.099189 (+ 2758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:52.099192 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:52.099206 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:52.099271 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:52.099277 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.100410 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.100413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:52.101437 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:52.101446 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.102257 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.102261 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:52.102362 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.102391 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 12:33:52.102469 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 12:33:52.102539 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260926 12:33:52.102712 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.096363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54632 (local address 127.10.98.1:45527)
0926 12:33:52.098776 (+ 2413us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:52.098786 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:52.098808 (+ 22us) server_negotiation.cc:407] Connection header received
0926 12:33:52.098874 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:52.098878 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:52.099047 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:52.099184 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:52.099394 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.100285 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.101551 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.102146 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.102245 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.102408 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:52.102499 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 12:33:52.102590 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2306,"thread_start_us":131,"threads_started":1}
I20260926 12:33:52.103318 18005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.097309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:33289 (local address 127.0.0.1:41448)
0926 12:33:52.098365 (+ 1056us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:52.098376 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:52.098493 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:52.099060 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:52.099062 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:52.099080 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:52.099204 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:52.099210 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.100760 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.100762 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:52.101908 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:52.101918 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.102961 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.102965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:52.103053 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.103077 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:33:52.103119 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 12:33:52.103182 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"thread_start_us":95,"threads_started":1}
I20260926 12:33:52.103984 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.097474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41448 (local address 127.10.98.2:33289)
0926 12:33:52.097783 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:52.097796 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:52.098484 (+ 688us) server_negotiation.cc:407] Connection header received
0926 12:33:52.098613 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:52.098620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:52.098878 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:52.099054 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:52.099338 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.100651 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.102054 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.102834 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.103574 (+ 740us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.103719 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:52.103763 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 12:33:52.103844 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":84,"threads_started":1}
I20260926 12:33:52.104866 18003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.096990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:46769 (local address 127.0.0.1:40012)
0926 12:33:52.097250 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:52.097264 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:52.097348 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:52.100705 (+ 3357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:52.100709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:52.100730 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:52.100834 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:52.100840 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.102424 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.102427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:52.103491 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:52.103501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.104527 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.104531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:52.104616 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.104639 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:33:52.104685 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 12:33:52.104745 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":4,"thread_start_us":107,"threads_started":1}
I20260926 12:33:52.105466 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.097176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40012 (local address 127.10.98.3:46769)
0926 12:33:52.099518 (+ 2342us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:52.099530 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:52.099632 (+ 102us) server_negotiation.cc:407] Connection header received
0926 12:33:52.099726 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:52.099731 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:52.099926 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:52.101085 (+ 1159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:52.101261 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.102308 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.103626 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.104392 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.105137 (+ 745us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.105257 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:52.105295 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 12:33:52.105361 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2192,"thread_start_us":87,"threads_started":1}
I20260926 12:33:52.106352 17635 tablet_service.cc:1467] Processing CreateTablet for tablet e60e46cf89d74f37a364a82588da6cbf (DEFAULT_TABLE table=test-table [id=8aaba92006b341aba419eb67f9015c72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:52.106722 17772 tablet_service.cc:1467] Processing CreateTablet for tablet e60e46cf89d74f37a364a82588da6cbf (DEFAULT_TABLE table=test-table [id=8aaba92006b341aba419eb67f9015c72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:52.107587 17635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e60e46cf89d74f37a364a82588da6cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:52.107646 17772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e60e46cf89d74f37a364a82588da6cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:52.107576 17908 tablet_service.cc:1467] Processing CreateTablet for tablet e60e46cf89d74f37a364a82588da6cbf (DEFAULT_TABLE table=test-table [id=8aaba92006b341aba419eb67f9015c72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:52.108400 17908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e60e46cf89d74f37a364a82588da6cbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:52.115155 18007 tablet_bootstrap.cc:492] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200: Bootstrap starting.
I20260926 12:33:52.116451 18008 tablet_bootstrap.cc:492] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc: Bootstrap starting.
I20260926 12:33:52.117400 18009 tablet_bootstrap.cc:492] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d: Bootstrap starting.
I20260926 12:33:52.117596 18007 tablet_bootstrap.cc:654] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:52.118479 18007 log.cc:824] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:52.118640 18008 tablet_bootstrap.cc:654] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:52.119230 18009 tablet_bootstrap.cc:654] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:52.119477 18008 log.cc:824] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:52.120160 18009 log.cc:824] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:52.120965 18007 tablet_bootstrap.cc:492] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200: No bootstrap required, opened a new log
I20260926 12:33:52.121202 18007 ts_tablet_manager.cc:1397] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:33:52.122072 18009 tablet_bootstrap.cc:492] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d: No bootstrap required, opened a new log
I20260926 12:33:52.122241 18009 ts_tablet_manager.cc:1397] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 12:33:52.125087 18007 raft_consensus.cc:348] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4f592a9801e467da0bc143fe5c41cbc" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45527 } } peers { permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 } } peers { permanent_uuid: "08fc683fa8894fa685caba488359483d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 33289 } }
I20260926 12:33:52.125383 18007 raft_consensus.cc:374] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:52.125442 18007 raft_consensus.cc:729] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46015c6e40794d6486049f51816f2200, State: Initialized, Role: FOLLOWER
I20260926 12:33:52.125929 18007 consensus_queue.cc:260] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4f592a9801e467da0bc143fe5c41cbc" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45527 } } peers { permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 } } peers { permanent_uuid: "08fc683fa8894fa685caba488359483d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 33289 } }
I20260926 12:33:52.125981 18009 raft_consensus.cc:348] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4f592a9801e467da0bc143fe5c41cbc" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45527 } } peers { permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 } } peers { permanent_uuid: "08fc683fa8894fa685caba488359483d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 33289 } }
I20260926 12:33:52.126267 18009 raft_consensus.cc:374] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:52.126317 18009 raft_consensus.cc:729] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08fc683fa8894fa685caba488359483d, State: Initialized, Role: FOLLOWER
I20260926 12:33:52.126757 18009 consensus_queue.cc:260] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4f592a9801e467da0bc143fe5c41cbc" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45527 } } peers { permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 } } peers { permanent_uuid: "08fc683fa8894fa685caba488359483d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 33289 } }
I20260926 12:33:52.127024 18007 ts_tablet_manager.cc:1428] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:33:52.127774 18009 ts_tablet_manager.cc:1428] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260926 12:33:52.128018 17975 heartbeater.cc:503] Master 127.10.98.62:40045 was elected leader, sending a full tablet report...
I20260926 12:33:52.128607 17839 heartbeater.cc:503] Master 127.10.98.62:40045 was elected leader, sending a full tablet report...
I20260926 12:33:52.127966 18008 tablet_bootstrap.cc:492] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc: No bootstrap required, opened a new log
I20260926 12:33:52.130425 18008 ts_tablet_manager.cc:1397] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260926 12:33:52.135216 18008 raft_consensus.cc:348] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4f592a9801e467da0bc143fe5c41cbc" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45527 } } peers { permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 } } peers { permanent_uuid: "08fc683fa8894fa685caba488359483d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 33289 } }
I20260926 12:33:52.135465 18008 raft_consensus.cc:374] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:52.135515 18008 raft_consensus.cc:729] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4f592a9801e467da0bc143fe5c41cbc, State: Initialized, Role: FOLLOWER
I20260926 12:33:52.135931 18008 consensus_queue.cc:260] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4f592a9801e467da0bc143fe5c41cbc" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45527 } } peers { permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 } } peers { permanent_uuid: "08fc683fa8894fa685caba488359483d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 33289 } }
I20260926 12:33:52.136619 17702 heartbeater.cc:503] Master 127.10.98.62:40045 was elected leader, sending a full tablet report...
I20260926 12:33:52.136987 18008 ts_tablet_manager.cc:1428] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260926 12:33:52.156196 17704 tablet.cc:2369] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:33:52.296980 17977 tablet.cc:2369] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:33:52.349074 17841 tablet.cc:2369] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:33:52.427161 18014 raft_consensus.cc:482] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:33:52.427410 18014 raft_consensus.cc:504] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4f592a9801e467da0bc143fe5c41cbc" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45527 } } peers { permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 } } peers { permanent_uuid: "08fc683fa8894fa685caba488359483d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 33289 } }
I20260926 12:33:52.428730 18014 leader_election.cc:288] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d4f592a9801e467da0bc143fe5c41cbc (127.10.98.1:45527), 46015c6e40794d6486049f51816f2200 (127.10.98.3:46769)
I20260926 12:33:52.432468 18002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.429047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:42937 (local address 127.10.98.1:45527)
0926 12:33:52.429221 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:52.429226 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:52.429448 (+ 222us) server_negotiation.cc:407] Connection header received
0926 12:33:52.429649 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:52.429652 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:52.429697 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:52.429759 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:52.430001 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.430476 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.431570 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.431952 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.432042 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.432106 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:52.432274 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 12:33:52.432344 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260926 12:33:52.432484 18016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.428985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:45527 (local address 127.10.98.2:42937)
0926 12:33:52.429360 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:52.429371 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:52.429538 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:52.429780 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:52.429783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:52.429797 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:52.429896 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:52.429902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.430608 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.430611 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:52.431451 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:52.431462 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.432070 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.432074 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:52.432167 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.432189 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:33:52.432253 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 12:33:52.432326 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":101,"threads_started":1}
I20260926 12:33:52.432665 18004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.429140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:46987 (local address 127.10.98.3:46769)
0926 12:33:52.429263 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:52.429267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:52.429415 (+ 148us) server_negotiation.cc:407] Connection header received
0926 12:33:52.429610 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:52.429612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:52.429656 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:52.429718 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:52.429949 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.430451 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.431681 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.432102 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.432198 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.432258 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:52.432478 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 12:33:52.432557 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:33:52.432680 18017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.428982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:46769 (local address 127.10.98.2:46987)
0926 12:33:52.429323 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:52.429334 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:52.429497 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:52.429734 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:52.429737 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:52.429752 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:52.429848 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:52.429854 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.430549 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.430552 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:52.431501 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:52.431511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.432225 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.432228 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:52.432323 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.432345 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:33:52.432407 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 12:33:52.432479 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":3,"thread_start_us":71,"threads_started":1}
I20260926 12:33:52.433286 17928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60e46cf89d74f37a364a82588da6cbf" candidate_uuid: "08fc683fa8894fa685caba488359483d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46015c6e40794d6486049f51816f2200" is_pre_election: true
I20260926 12:33:52.433284 17655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60e46cf89d74f37a364a82588da6cbf" candidate_uuid: "08fc683fa8894fa685caba488359483d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4f592a9801e467da0bc143fe5c41cbc" is_pre_election: true
I20260926 12:33:52.433584 17655 raft_consensus.cc:2454] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08fc683fa8894fa685caba488359483d in term 0.
I20260926 12:33:52.433588 17928 raft_consensus.cc:2454] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08fc683fa8894fa685caba488359483d in term 0.
I20260926 12:33:52.434047 17725 leader_election.cc:302] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08fc683fa8894fa685caba488359483d, d4f592a9801e467da0bc143fe5c41cbc; no voters:
I20260926 12:33:52.434315 18014 raft_consensus.cc:2781] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:33:52.434406 18014 raft_consensus.cc:482] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:33:52.434464 18014 raft_consensus.cc:3037] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:52.435499 18014 raft_consensus.cc:504] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4f592a9801e467da0bc143fe5c41cbc" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45527 } } peers { permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 } } peers { permanent_uuid: "08fc683fa8894fa685caba488359483d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 33289 } }
I20260926 12:33:52.435822 18014 leader_election.cc:288] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d [CANDIDATE]: Term 1 election: Requested vote from peers d4f592a9801e467da0bc143fe5c41cbc (127.10.98.1:45527), 46015c6e40794d6486049f51816f2200 (127.10.98.3:46769)
I20260926 12:33:52.436157 17928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60e46cf89d74f37a364a82588da6cbf" candidate_uuid: "08fc683fa8894fa685caba488359483d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46015c6e40794d6486049f51816f2200"
I20260926 12:33:52.436165 17655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e60e46cf89d74f37a364a82588da6cbf" candidate_uuid: "08fc683fa8894fa685caba488359483d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d4f592a9801e467da0bc143fe5c41cbc"
I20260926 12:33:52.436266 17928 raft_consensus.cc:3037] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:52.436277 17655 raft_consensus.cc:3037] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:52.437314 17928 raft_consensus.cc:2454] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08fc683fa8894fa685caba488359483d in term 1.
I20260926 12:33:52.437331 17655 raft_consensus.cc:2454] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08fc683fa8894fa685caba488359483d in term 1.
I20260926 12:33:52.437558 17726 leader_election.cc:302] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08fc683fa8894fa685caba488359483d, 46015c6e40794d6486049f51816f2200; no voters:
I20260926 12:33:52.437754 18014 raft_consensus.cc:2781] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:52.437943 18014 raft_consensus.cc:686] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d [term 1 LEADER]: Becoming Leader. State: Replica: 08fc683fa8894fa685caba488359483d, State: Running, Role: LEADER
I20260926 12:33:52.438364 18014 consensus_queue.cc:237] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4f592a9801e467da0bc143fe5c41cbc" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45527 } } peers { permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 } } peers { permanent_uuid: "08fc683fa8894fa685caba488359483d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 33289 } }
I20260926 12:33:52.442174 16738 catalog_manager.cc:5495] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d reported cstate change: term changed from 0 to 1, leader changed from <none> to 08fc683fa8894fa685caba488359483d (127.10.98.2). New cstate: current_term: 1 leader_uuid: "08fc683fa8894fa685caba488359483d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d4f592a9801e467da0bc143fe5c41cbc" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "08fc683fa8894fa685caba488359483d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 33289 } health_report { overall_health: HEALTHY } } }
I20260926 12:33:52.471287 18006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.467442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41462 (local address 127.10.98.2:33289)
0926 12:33:52.467553 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:52.467583 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:52.467597 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:33:52.467673 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:52.467676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:52.467733 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:52.467822 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:52.468118 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.468740 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.469679 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.469954 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.470054 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.470423 (+ 369us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:52.470898 (+ 475us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:52.470993 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:52.471125 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 12:33:52.471179 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 12:33:52.477384 17928 raft_consensus.cc:1261] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200 [term 1 FOLLOWER]: Refusing update from remote peer 08fc683fa8894fa685caba488359483d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:52.477402 17655 raft_consensus.cc:1261] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc [term 1 FOLLOWER]: Refusing update from remote peer 08fc683fa8894fa685caba488359483d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:52.478137 18018 consensus_queue.cc:1035] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d [LEADER]: Connected to new peer: Peer: permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:52.478304 18014 consensus_queue.cc:1035] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d4f592a9801e467da0bc143fe5c41cbc" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:52.484737 18021 mvcc.cc:205] Tried to move back new op lower bound from 7333585029019631616 to 7333585028870189056. Current Snapshot: MvccSnapshot[applied={T|T < 7333585029019631616}]
I20260926 12:33:52.487020 18023 mvcc.cc:205] Tried to move back new op lower bound from 7333585029019631616 to 7333585028870189056. Current Snapshot: MvccSnapshot[applied={T|T < 7333585029019631616}]
I20260926 12:33:52.485829 18022 mvcc.cc:205] Tried to move back new op lower bound from 7333585029019631616 to 7333585028870189056. Current Snapshot: MvccSnapshot[applied={T|T < 7333585029019631616}]
I20260926 12:33:52.499950 16738 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:52.500142 16738 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53714:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:52.504489 16738 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e1e1ab3339074143aa34a2cbe581dc5b: Sending DeleteTablet for 3 replicas of tablet e60e46cf89d74f37a364a82588da6cbf
I20260926 12:33:52.505324 17635 tablet_service.cc:1514] Processing DeleteTablet for tablet e60e46cf89d74f37a364a82588da6cbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:52 UTC) from {username='oryx', principal='oryx/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:54632
I20260926 12:33:52.505532 17772 tablet_service.cc:1514] Processing DeleteTablet for tablet e60e46cf89d74f37a364a82588da6cbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:52 UTC) from {username='oryx', principal='oryx/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:41448
I20260926 12:33:52.505553 17908 tablet_service.cc:1514] Processing DeleteTablet for tablet e60e46cf89d74f37a364a82588da6cbf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:52 UTC) from {username='oryx', principal='oryx/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:40012
I20260926 12:33:52.506027 18032 tablet_replica.cc:330] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200: stopping tablet replica
I20260926 12:33:52.506033 18033 tablet_replica.cc:330] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d: stopping tablet replica
I20260926 12:33:52.506410 18032 raft_consensus.cc:2229] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:52.506414 18033 raft_consensus.cc:2229] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:33:52.506783 18032 raft_consensus.cc:2258] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:52.507001 18033 raft_consensus.cc:2258] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:52.508744 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 16707
I20260926 12:33:52.508726 18032 ts_tablet_manager.cc:1905] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:52.508726 18033 ts_tablet_manager.cc:1905] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:52.511682 18032 ts_tablet_manager.cc:1918] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:33:52.511786 18032 log.cc:1196] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/e60e46cf89d74f37a364a82588da6cbf
I20260926 12:33:52.512122 18032 ts_tablet_manager.cc:1939] T e60e46cf89d74f37a364a82588da6cbf P 46015c6e40794d6486049f51816f2200: Deleting consensus metadata
I20260926 12:33:52.512494 18033 ts_tablet_manager.cc:1918] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:33:52.512645 18033 log.cc:1196] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/e60e46cf89d74f37a364a82588da6cbf
I20260926 12:33:52.512991 18033 ts_tablet_manager.cc:1939] T e60e46cf89d74f37a364a82588da6cbf P 08fc683fa8894fa685caba488359483d: Deleting consensus metadata
I20260926 12:33:52.515069 18031 tablet_replica.cc:330] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc: stopping tablet replica
I20260926 12:33:52.515419 18031 raft_consensus.cc:2229] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:52.515743 18031 raft_consensus.cc:2258] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 12:33:52.516113 17726 connection.cc:535] server connection from 127.0.0.1:41448 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 12:33:52.516131 17862 connection.cc:535] server connection from 127.0.0.1:40012 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 12:33:52.516731 17509 connection.cc:535] server connection from 127.0.0.1:35894 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 12:33:52.517681 18031 ts_tablet_manager.cc:1905] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:52.520323 18031 ts_tablet_manager.cc:1918] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:33:52.520423 18031 log.cc:1196] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-0/wal/wals/e60e46cf89d74f37a364a82588da6cbf
I20260926 12:33:52.520764 18031 ts_tablet_manager.cc:1939] T e60e46cf89d74f37a364a82588da6cbf P d4f592a9801e467da0bc143fe5c41cbc: Deleting consensus metadata
W20260926 12:33:52.522212 17588 connection.cc:441] server connection from 127.0.0.1:54632 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54632 (request call id 1) could send its response
I20260926 12:33:52.525329 17989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.507658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49884 (local address 127.10.98.61:38279)
0926 12:33:52.507770 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:52.507775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:52.513619 (+ 5844us) server_negotiation.cc:407] Connection header received
0926 12:33:52.513812 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:52.513816 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:52.513870 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:52.513919 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:52.514918 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.515667 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.518020 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.518366 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.518452 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.521223 (+ 2771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:52.521227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:52.521230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:52.521258 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:52.523210 (+ 1952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:52.523851 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:52.523854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:52.523856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:52.523948 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:52.524260 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:52.524264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:52.524266 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:52.524574 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:52.524691 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:52.525025 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 12:33:52.525172 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:33:52.532027 17705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.520089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35924 (local address 127.10.98.60:38335)
0926 12:33:52.520214 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:52.520219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:52.520853 (+ 634us) server_negotiation.cc:407] Connection header received
0926 12:33:52.521036 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:52.521039 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:52.521090 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:52.521168 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:52.522603 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.523414 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.524894 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.525222 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.525295 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.527028 (+ 1733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:52.527032 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:52.527034 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:52.527062 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:52.528791 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:52.530721 (+ 1930us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:52.530725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:52.530727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:52.530810 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:52.531100 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:52.531103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:52.531104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:52.531308 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:52.531408 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:52.531750 (+ 342us) server_negotiation.cc:299] Negotiation successful
0926 12:33:52.531866 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:33:52.532276 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.523345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49886 (local address 127.10.98.61:38279)
0926 12:33:52.523477 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:52.523482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:52.523864 (+ 382us) server_negotiation.cc:407] Connection header received
0926 12:33:52.524039 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:52.524043 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:52.524104 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:52.524153 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:52.525009 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.525820 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.527085 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.527416 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.527480 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.529038 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:52.529042 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:52.529045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:52.529069 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:52.530615 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:52.531026 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:52.531030 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:52.531032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:52.531110 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:52.531400 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:52.531403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:52.531404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:52.531596 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:52.531691 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:52.532035 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 12:33:52.532143 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:33:52.536187 17990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:52.507327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35908 (local address 127.10.98.60:38335)
0926 12:33:52.507756 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:52.507761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:52.527259 (+ 19498us) server_negotiation.cc:407] Connection header received
0926 12:33:52.527754 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:52.527757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:52.527811 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:52.527875 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:52.528665 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.529525 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.530640 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:52.531003 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:52.531099 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:52.532772 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:52.532776 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:52.532778 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:52.532802 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:52.534363 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:52.534755 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:52.534758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:52.534760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:52.534838 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:52.535281 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:52.535284 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:52.535286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:52.535485 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:52.535598 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:52.535905 (+ 307us) server_negotiation.cc:299] Negotiation successful
0926 12:33:52.536031 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343}
I20260926 12:33:53.490262 18041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:53.489726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.2:37931)
0926 12:33:53.490050 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:53.490147 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":79,"threads_started":1}
I20260926 12:33:53.490712 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:53.490233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.1:33517)
0926 12:33:53.490528 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:53.490620 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":70,"threads_started":1}
W20260926 12:33:53.490705 17839 heartbeater.cc:644] Failed to heartbeat to 127.10.98.62:40045 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
W20260926 12:33:53.491020 17702 heartbeater.cc:644] Failed to heartbeat to 127.10.98.62:40045 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
I20260926 12:33:53.492378 18043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:53.492042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.3:38653)
0926 12:33:53.492237 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:53.492304 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":57,"threads_started":1}
W20260926 12:33:53.492638 17975 heartbeater.cc:644] Failed to heartbeat to 127.10.98.62:40045 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
I20260926 12:33:54.004976 18044 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1e1ab3339074143aa34a2cbe581dc5b)
I20260926 12:33:54.005226 18044 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false } }
I20260926 12:33:54.005342 18045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e1e1ab3339074143aa34a2cbe581dc5b)
I20260926 12:33:54.005499 18045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false } }
I20260926 12:33:54.006379 18044 leader_election.cc:288] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1e1ab3339074143aa34a2cbe581dc5b (127.10.98.62:40045), 08b7010c899c498e9e6af4db7d5e95d2 (127.10.98.61:38279)
I20260926 12:33:54.006760 18045 leader_election.cc:288] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e1e1ab3339074143aa34a2cbe581dc5b (127.10.98.62:40045), fd967af90a354d40bcf567af49879f6f (127.10.98.60:38335)
I20260926 12:33:54.007068 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:54.006574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.0.0.1:54188)
0926 12:33:54.006865 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:54.006954 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":60,"threads_started":1}
I20260926 12:33:54.007280 18048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:54.006554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.0.0.1:54186)
0926 12:33:54.007129 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:54.007192 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":76,"threads_started":1}
W20260926 12:33:54.007294 17507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
W20260926 12:33:54.007431 17324 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
I20260926 12:33:54.008213 18046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:54.007912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.0.0.1:54196)
0926 12:33:54.008094 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:54.008147 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260926 12:33:54.008318 18048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:54.008088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.0.0.1:54204)
0926 12:33:54.008212 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:54.008254 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260926 12:33:54.008355 17507 leader_election.cc:334] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e1e1ab3339074143aa34a2cbe581dc5b (127.10.98.62:40045): Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
W20260926 12:33:54.008443 17324 leader_election.cc:334] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer e1e1ab3339074143aa34a2cbe581dc5b (127.10.98.62:40045): Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
I20260926 12:33:54.011544 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:54.006936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49898 (local address 127.10.98.61:38279)
0926 12:33:54.007306 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:54.007311 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:54.007326 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:33:54.007373 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:54.007376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:54.007420 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:54.007510 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:54.007858 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:54.008765 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:54.010312 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:54.010942 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:54.011046 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:54.011120 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:54.011305 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 12:33:54.011418 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":308,"thread_start_us":46,"threads_started":1}
I20260926 12:33:54.011549 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:54.006886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:38279 (local address 127.0.0.1:49898)
0926 12:33:54.007049 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:54.007061 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:54.007175 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:54.007591 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:54.007594 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:54.007616 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:54.007727 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:54.007734 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:54.008860 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:54.008863 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:54.010197 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:54.010207 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:54.011061 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:54.011064 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:54.011170 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:54.011210 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 12:33:54.011283 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 12:33:54.011364 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":4,"thread_start_us":48,"threads_started":1}
I20260926 12:33:54.011556 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:54.006734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35928 (local address 127.10.98.60:38335)
0926 12:33:54.006958 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:54.006963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:54.007780 (+ 817us) server_negotiation.cc:407] Connection header received
0926 12:33:54.007952 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:54.007955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:54.007997 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:54.008048 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:54.008395 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:54.009002 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:54.010468 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:54.011056 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:54.011205 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:54.011247 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:54.011343 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 12:33:54.011415 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":157,"thread_start_us":58,"threads_started":1}
I20260926 12:33:54.011588 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:54.006582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:38335 (local address 127.0.0.1:35928)
0926 12:33:54.007709 (+ 1127us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:54.007722 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:54.007845 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:54.008174 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:54.008177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:54.008195 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:54.008275 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:54.008282 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:54.009162 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:54.009165 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:54.010367 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:54.010375 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:54.011157 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:54.011161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:54.011241 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:54.011262 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:33:54.011333 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 12:33:54.011445 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1070,"mutex_wait_us":45,"thread_start_us":57,"threads_started":1}
I20260926 12:33:54.012090 17349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd967af90a354d40bcf567af49879f6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "08b7010c899c498e9e6af4db7d5e95d2" is_pre_election: true
I20260926 12:33:54.012255 17532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08b7010c899c498e9e6af4db7d5e95d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fd967af90a354d40bcf567af49879f6f" is_pre_election: true
I20260926 12:33:54.012359 17349 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd967af90a354d40bcf567af49879f6f in term 1.
I20260926 12:33:54.012408 17532 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08b7010c899c498e9e6af4db7d5e95d2 in term 1.
I20260926 12:33:54.012686 17507 leader_election.cc:302] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08b7010c899c498e9e6af4db7d5e95d2, fd967af90a354d40bcf567af49879f6f; no voters: e1e1ab3339074143aa34a2cbe581dc5b
I20260926 12:33:54.012688 17326 leader_election.cc:302] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08b7010c899c498e9e6af4db7d5e95d2, fd967af90a354d40bcf567af49879f6f; no voters: e1e1ab3339074143aa34a2cbe581dc5b
I20260926 12:33:54.012912 18044 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 12:33:54.012971 18044 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 1 FOLLOWER]: Starting leader election (detected failure of leader e1e1ab3339074143aa34a2cbe581dc5b)
I20260926 12:33:54.013015 18044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 1 FOLLOWER]: Advancing to term 2
I20260926 12:33:54.013186 18045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 12:33:54.013243 18045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e1e1ab3339074143aa34a2cbe581dc5b)
I20260926 12:33:54.013286 18045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 1 FOLLOWER]: Advancing to term 2
I20260926 12:33:54.014374 18045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false } }
I20260926 12:33:54.014374 18044 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false } }
I20260926 12:33:54.014761 18044 leader_election.cc:288] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [CANDIDATE]: Term 2 election: Requested vote from peers e1e1ab3339074143aa34a2cbe581dc5b (127.10.98.62:40045), 08b7010c899c498e9e6af4db7d5e95d2 (127.10.98.61:38279)
I20260926 12:33:54.014770 18045 leader_election.cc:288] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [CANDIDATE]: Term 2 election: Requested vote from peers e1e1ab3339074143aa34a2cbe581dc5b (127.10.98.62:40045), fd967af90a354d40bcf567af49879f6f (127.10.98.60:38335)
I20260926 12:33:54.015107 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:54.014893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.0.0.1:54214)
0926 12:33:54.014987 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:54.015042 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 12:33:54.015107 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:54.014893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.0.0.1:54216)
0926 12:33:54.015003 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:54.015052 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 12:33:54.015237 17532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08b7010c899c498e9e6af4db7d5e95d2" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fd967af90a354d40bcf567af49879f6f"
I20260926 12:33:54.015404 17532 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 08b7010c899c498e9e6af4db7d5e95d2 in current term 2: Already voted for candidate fd967af90a354d40bcf567af49879f6f in this term.
I20260926 12:33:54.015607 17349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd967af90a354d40bcf567af49879f6f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "08b7010c899c498e9e6af4db7d5e95d2"
I20260926 12:33:54.015753 17349 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fd967af90a354d40bcf567af49879f6f in current term 2: Already voted for candidate 08b7010c899c498e9e6af4db7d5e95d2 in this term.
I20260926 12:33:54.015837 18047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:54.015504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.0.0.1:54242)
0926 12:33:54.015679 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:54.015745 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260926 12:33:54.015841 18050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:54.015619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.0.0.1:54244)
0926 12:33:54.015727 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:54.015782 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 12:33:54.015928 17507 leader_election.cc:334] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e1e1ab3339074143aa34a2cbe581dc5b (127.10.98.62:40045): Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
W20260926 12:33:54.016041 17324 leader_election.cc:334] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer e1e1ab3339074143aa34a2cbe581dc5b (127.10.98.62:40045): Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
I20260926 12:33:54.016124 17324 leader_election.cc:302] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 08b7010c899c498e9e6af4db7d5e95d2; no voters: e1e1ab3339074143aa34a2cbe581dc5b, fd967af90a354d40bcf567af49879f6f
I20260926 12:33:54.016232 18045 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 12:33:54.016458 17507 leader_election.cc:302] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd967af90a354d40bcf567af49879f6f; no voters: 08b7010c899c498e9e6af4db7d5e95d2, e1e1ab3339074143aa34a2cbe581dc5b
I20260926 12:33:54.016597 18044 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 12:33:54.491896 18054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:54.491291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.2:37723)
0926 12:33:54.491715 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:54.491782 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":105,"threads_started":1}
I20260926 12:33:54.491928 18055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:54.491489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.1:53851)
0926 12:33:54.491770 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:54.491832 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":77,"threads_started":1}
I20260926 12:33:54.493530 18056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:54.493186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.3:32955)
0926 12:33:54.493394 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:54.493443 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":63,"threads_started":1}
I20260926 12:33:55.493744 18057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:55.492789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.1:35613)
0926 12:33:55.493476 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:55.493627 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":74,"threads_started":1}
I20260926 12:33:55.493961 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:55.493319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.2:50979)
0926 12:33:55.493662 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:55.493727 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":64,"threads_started":1}
I20260926 12:33:55.494748 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:55.494222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.3:56293)
0926 12:33:55.494605 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:55.494663 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":88,"threads_started":1}
I20260926 12:33:55.701673 18060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:33:55.701813 18060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false } }
I20260926 12:33:55.702314 18060 leader_election.cc:288] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e1e1ab3339074143aa34a2cbe581dc5b (127.10.98.62:40045), fd967af90a354d40bcf567af49879f6f (127.10.98.60:38335)
I20260926 12:33:55.702858 17532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08b7010c899c498e9e6af4db7d5e95d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fd967af90a354d40bcf567af49879f6f" is_pre_election: true
I20260926 12:33:55.702936 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:55.702513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.0.0.1:54492)
0926 12:33:55.702795 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:55.702852 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":72,"threads_started":1}
I20260926 12:33:55.703167 17532 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 08b7010c899c498e9e6af4db7d5e95d2 in term 2.
I20260926 12:33:55.703598 17326 leader_election.cc:302] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08b7010c899c498e9e6af4db7d5e95d2, fd967af90a354d40bcf567af49879f6f; no voters:
I20260926 12:33:55.703756 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:55.703529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.0.0.1:54498)
0926 12:33:55.703640 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:55.703693 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 12:33:55.703760 18060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 12:33:55.703836 18060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260926 12:33:55.703876 17324 leader_election.cc:334] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer e1e1ab3339074143aa34a2cbe581dc5b (127.10.98.62:40045): Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
I20260926 12:33:55.703903 18060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 2 FOLLOWER]: Advancing to term 3
I20260926 12:33:55.704998 18060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false } }
I20260926 12:33:55.705348 18060 leader_election.cc:288] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [CANDIDATE]: Term 3 election: Requested vote from peers e1e1ab3339074143aa34a2cbe581dc5b (127.10.98.62:40045), fd967af90a354d40bcf567af49879f6f (127.10.98.60:38335)
I20260926 12:33:55.705664 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:55.705459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.0.0.1:54510)
0926 12:33:55.705571 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:55.705613 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 12:33:55.705680 17532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "08b7010c899c498e9e6af4db7d5e95d2" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fd967af90a354d40bcf567af49879f6f"
I20260926 12:33:55.705801 17532 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 2 FOLLOWER]: Advancing to term 3
I20260926 12:33:55.706262 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:55.706086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.0.0.1:54526)
0926 12:33:55.706176 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:55.706216 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 12:33:55.706354 17324 leader_election.cc:334] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer e1e1ab3339074143aa34a2cbe581dc5b (127.10.98.62:40045): Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
I20260926 12:33:55.706921 17532 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 08b7010c899c498e9e6af4db7d5e95d2 in term 3.
I20260926 12:33:55.707202 17326 leader_election.cc:302] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 08b7010c899c498e9e6af4db7d5e95d2, fd967af90a354d40bcf567af49879f6f; no voters: e1e1ab3339074143aa34a2cbe581dc5b
I20260926 12:33:55.707371 18060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 12:33:55.707532 18060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [term 3 LEADER]: Becoming Leader. State: Replica: 08b7010c899c498e9e6af4db7d5e95d2, State: Running, Role: LEADER
I20260926 12:33:55.707773 18060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false } }
I20260926 12:33:55.709164 18063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08b7010c899c498e9e6af4db7d5e95d2. Latest consensus state: current_term: 3 leader_uuid: "08b7010c899c498e9e6af4db7d5e95d2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false } } }
I20260926 12:33:55.709290 18063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:55.709563 18065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:33:55.711750 18065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:33:55.712105 18065 catalog_manager.cc:1259] Loaded cluster ID: f47659f98ed746639f42fd35d409569e
I20260926 12:33:55.712149 18065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:33:55.712666 18065 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:33:55.713001 18065 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: Loaded TSK: 0
I20260926 12:33:55.713693 18065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:33:55.760491 17338 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:33:55.763705 17338 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:33:55.766993 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:55.766757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.0.0.1:54548)
0926 12:33:55.766866 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:55.766921 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 12:33:55.767172 17532 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [term 3 FOLLOWER]: Refusing update from remote peer 08b7010c899c498e9e6af4db7d5e95d2: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 12:33:55.767665 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:55.767432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.0.0.1:54562)
0926 12:33:55.767527 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:55.767597 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 12:33:55.767679 18063 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260926 12:33:55.767858 17324 consensus_peers.cc:487] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 -> Peer e1e1ab3339074143aa34a2cbe581dc5b (127.10.98.62:40045): Couldn't send request to peer e1e1ab3339074143aa34a2cbe581dc5b. Status: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 12:33:55.769244 18067 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08b7010c899c498e9e6af4db7d5e95d2. Latest consensus state: current_term: 3 leader_uuid: "08b7010c899c498e9e6af4db7d5e95d2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false } } }
I20260926 12:33:55.769388 18067 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:33:55.770444 18060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "08b7010c899c498e9e6af4db7d5e95d2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false } } }
I20260926 12:33:55.770565 18060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:55.770844 18067 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "08b7010c899c498e9e6af4db7d5e95d2" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "e1e1ab3339074143aa34a2cbe581dc5b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40045 } } peers { permanent_uuid: "08b7010c899c498e9e6af4db7d5e95d2" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 38279 } attrs { promote: false } } peers { permanent_uuid: "fd967af90a354d40bcf567af49879f6f" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 38335 } attrs { promote: false } } }
I20260926 12:33:55.770967 18067 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd967af90a354d40bcf567af49879f6f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:33:55.781452 18074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:55.776896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:45527 (local address 127.0.0.1:54644)
0926 12:33:55.777352 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:55.777365 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:55.777448 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:55.777900 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:55.777903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:55.777917 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:55.778025 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:55.778031 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:55.778930 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:55.778934 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:55.780122 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:55.780132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:55.781063 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:55.781067 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:55.781179 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:55.781200 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:33:55.781247 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 12:33:55.781322 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":1,"thread_start_us":95,"threads_started":1}
I20260926 12:33:55.781630 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:55.776713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:46769 (local address 127.0.0.1:40016)
0926 12:33:55.776864 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:55.776878 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:55.776970 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:55.778489 (+ 1519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:55.778491 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:55.778508 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:55.778579 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:55.778584 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:55.779480 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:55.779483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:55.780350 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:55.780360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:55.781216 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:55.781219 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:55.781325 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:55.781346 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:33:55.781422 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 12:33:55.781501 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":54}
I20260926 12:33:55.781634 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:55.776808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40016 (local address 127.10.98.3:46769)
0926 12:33:55.778263 (+ 1455us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:55.778267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:55.778279 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:33:55.778328 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:55.778331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:55.778376 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:55.778463 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:55.778707 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:55.779380 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:55.780480 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:55.781076 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:55.781298 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:55.781360 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:55.781436 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 12:33:55.781505 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1386,"thread_start_us":93,"threads_started":1}
I20260926 12:33:55.782081 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:55.777474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54644 (local address 127.10.98.1:45527)
0926 12:33:55.777697 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:55.777701 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:55.777714 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:33:55.777763 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:55.777766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:55.777813 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:55.777895 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:55.778118 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:55.778836 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:55.780258 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:55.780901 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:55.781751 (+ 850us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:55.781817 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:55.781857 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 12:33:55.781935 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":69,"threads_started":1}
I20260926 12:33:55.782164 18075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:55.777210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:33289 (local address 127.0.0.1:41464)
0926 12:33:55.777467 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:55.777488 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:55.777560 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:55.778139 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:55.778142 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:55.778159 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:55.778226 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:55.778231 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:55.779182 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:55.779185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:55.780303 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:55.780310 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:55.781189 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:55.781193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:55.781875 (+ 682us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:55.781898 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:33:55.781964 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 12:33:55.782044 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260926 12:33:55.782208 17908 tablet_service.cc:1467] Processing CreateTablet for tablet d454455a9b154586b0ef7644e5941dd1 (DEFAULT_TABLE table=test-table [id=d4d7c77d630f4a32a33526f01713142f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:55.782370 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:55.777318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41464 (local address 127.10.98.2:33289)
0926 12:33:55.777838 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:55.777843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:55.777858 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:33:55.777912 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:55.777915 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:55.777964 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:55.778052 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:55.778330 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:55.779073 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:55.780422 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:55.781050 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:55.781168 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:55.781242 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:55.782000 (+ 758us) server_negotiation.cc:299] Negotiation successful
0926 12:33:55.782093 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":58,"threads_started":1}
I20260926 12:33:55.782521 17908 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d454455a9b154586b0ef7644e5941dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:55.782548 17635 tablet_service.cc:1467] Processing CreateTablet for tablet d454455a9b154586b0ef7644e5941dd1 (DEFAULT_TABLE table=test-table [id=d4d7c77d630f4a32a33526f01713142f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:55.782783 17635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d454455a9b154586b0ef7644e5941dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:55.784766 18078 tablet_bootstrap.cc:492] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc: Bootstrap starting.
I20260926 12:33:55.784909 18079 tablet_bootstrap.cc:492] T d454455a9b154586b0ef7644e5941dd1 P 46015c6e40794d6486049f51816f2200: Bootstrap starting.
I20260926 12:33:55.785261 17772 tablet_service.cc:1467] Processing CreateTablet for tablet d454455a9b154586b0ef7644e5941dd1 (DEFAULT_TABLE table=test-table [id=d4d7c77d630f4a32a33526f01713142f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:55.785531 17772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d454455a9b154586b0ef7644e5941dd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:55.786586 18078 tablet_bootstrap.cc:654] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:55.787678 18078 tablet_bootstrap.cc:492] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc: No bootstrap required, opened a new log
I20260926 12:33:55.787772 18078 ts_tablet_manager.cc:1397] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 12:33:55.787882 18081 tablet_bootstrap.cc:492] T d454455a9b154586b0ef7644e5941dd1 P 08fc683fa8894fa685caba488359483d: Bootstrap starting.
I20260926 12:33:55.788223 18078 raft_consensus.cc:348] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 } } peers { permanent_uuid: "d4f592a9801e467da0bc143fe5c41cbc" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45527 } } peers { permanent_uuid: "08fc683fa8894fa685caba488359483d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 33289 } }
I20260926 12:33:55.788347 18078 raft_consensus.cc:374] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:55.788380 18078 raft_consensus.cc:729] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d4f592a9801e467da0bc143fe5c41cbc, State: Initialized, Role: FOLLOWER
I20260926 12:33:55.788542 18079 tablet_bootstrap.cc:654] T d454455a9b154586b0ef7644e5941dd1 P 46015c6e40794d6486049f51816f2200: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:55.788547 18078 consensus_queue.cc:260] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 } } peers { permanent_uuid: "d4f592a9801e467da0bc143fe5c41cbc" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45527 } } peers { permanent_uuid: "08fc683fa8894fa685caba488359483d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 33289 } }
I20260926 12:33:55.788735 18081 tablet_bootstrap.cc:654] T d454455a9b154586b0ef7644e5941dd1 P 08fc683fa8894fa685caba488359483d: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:55.788926 18078 ts_tablet_manager.cc:1428] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:55.789752 18079 tablet_bootstrap.cc:492] T d454455a9b154586b0ef7644e5941dd1 P 46015c6e40794d6486049f51816f2200: No bootstrap required, opened a new log
I20260926 12:33:55.789826 18057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:55.789597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.1:46183)
0926 12:33:55.789699 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:55.789755 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 12:33:55.789839 18079 ts_tablet_manager.cc:1397] T d454455a9b154586b0ef7644e5941dd1 P 46015c6e40794d6486049f51816f2200: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 12:33:55.790310 18079 raft_consensus.cc:348] T d454455a9b154586b0ef7644e5941dd1 P 46015c6e40794d6486049f51816f2200 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 } } peers { permanent_uuid: "d4f592a9801e467da0bc143fe5c41cbc" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45527 } } peers { permanent_uuid: "08fc683fa8894fa685caba488359483d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 33289 } }
I20260926 12:33:55.790416 18079 raft_consensus.cc:374] T d454455a9b154586b0ef7644e5941dd1 P 46015c6e40794d6486049f51816f2200 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:55.790446 18079 raft_consensus.cc:729] T d454455a9b154586b0ef7644e5941dd1 P 46015c6e40794d6486049f51816f2200 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46015c6e40794d6486049f51816f2200, State: Initialized, Role: FOLLOWER
I20260926 12:33:55.790594 18079 consensus_queue.cc:260] T d454455a9b154586b0ef7644e5941dd1 P 46015c6e40794d6486049f51816f2200 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 } } peers { permanent_uuid: "d4f592a9801e467da0bc143fe5c41cbc" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45527 } } peers { permanent_uuid: "08fc683fa8894fa685caba488359483d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 33289 } }
I20260926 12:33:55.790711 18057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:55.790442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.1:39185)
0926 12:33:55.790601 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:55.790649 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 12:33:55.790948 18079 ts_tablet_manager.cc:1428] T d454455a9b154586b0ef7644e5941dd1 P 46015c6e40794d6486049f51816f2200: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:55.791898 17703 heartbeater.cc:503] Master 127.10.98.61:38279 was elected leader, sending a full tablet report...
I20260926 12:33:55.792162 18081 tablet_bootstrap.cc:492] T d454455a9b154586b0ef7644e5941dd1 P 08fc683fa8894fa685caba488359483d: No bootstrap required, opened a new log
I20260926 12:33:55.792265 18081 ts_tablet_manager.cc:1397] T d454455a9b154586b0ef7644e5941dd1 P 08fc683fa8894fa685caba488359483d: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 12:33:55.792354 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:55.791807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.3:60583)
0926 12:33:55.792231 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:55.792288 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348}
I20260926 12:33:55.792798 18081 raft_consensus.cc:348] T d454455a9b154586b0ef7644e5941dd1 P 08fc683fa8894fa685caba488359483d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 } } peers { permanent_uuid: "d4f592a9801e467da0bc143fe5c41cbc" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45527 } } peers { permanent_uuid: "08fc683fa8894fa685caba488359483d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 33289 } }
I20260926 12:33:55.792922 18081 raft_consensus.cc:374] T d454455a9b154586b0ef7644e5941dd1 P 08fc683fa8894fa685caba488359483d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:55.792941 17976 heartbeater.cc:503] Master 127.10.98.61:38279 was elected leader, sending a full tablet report...
I20260926 12:33:55.792963 18081 raft_consensus.cc:729] T d454455a9b154586b0ef7644e5941dd1 P 08fc683fa8894fa685caba488359483d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08fc683fa8894fa685caba488359483d, State: Initialized, Role: FOLLOWER
I20260926 12:33:55.793128 18081 consensus_queue.cc:260] T d454455a9b154586b0ef7644e5941dd1 P 08fc683fa8894fa685caba488359483d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 } } peers { permanent_uuid: "d4f592a9801e467da0bc143fe5c41cbc" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45527 } } peers { permanent_uuid: "08fc683fa8894fa685caba488359483d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 33289 } }
I20260926 12:33:55.793498 18081 ts_tablet_manager.cc:1428] T d454455a9b154586b0ef7644e5941dd1 P 08fc683fa8894fa685caba488359483d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260926 12:33:55.794252 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:55.793918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.2:39311)
0926 12:33:55.794133 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:55.794185 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
I20260926 12:33:55.795303 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:55.795092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.2:39479)
0926 12:33:55.795190 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:55.795242 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 12:33:55.795441 17840 heartbeater.cc:503] Master 127.10.98.61:38279 was elected leader, sending a full tablet report...
I20260926 12:33:55.796080 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:55.795834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.3:37957)
0926 12:33:55.795971 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:55.796020 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260926 12:33:56.060635 18082 raft_consensus.cc:482] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:33:56.060861 18082 raft_consensus.cc:504] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 } } peers { permanent_uuid: "d4f592a9801e467da0bc143fe5c41cbc" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45527 } } peers { permanent_uuid: "08fc683fa8894fa685caba488359483d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 33289 } }
I20260926 12:33:56.061996 18082 leader_election.cc:288] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46015c6e40794d6486049f51816f2200 (127.10.98.3:46769), 08fc683fa8894fa685caba488359483d (127.10.98.2:33289)
I20260926 12:33:56.066066 18057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:56.062295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:46769 (local address 127.10.98.1:58569)
0926 12:33:56.062430 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:56.062445 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:56.062547 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:56.062927 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:56.062930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:56.062948 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:56.063064 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:56.063071 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:56.063982 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:56.063986 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:56.064882 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:56.064892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:56.065647 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:56.065651 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:56.065761 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:56.065787 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 12:33:56.065860 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 12:33:56.065926 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":26}
I20260926 12:33:56.066184 18073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:56.062398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:58569 (local address 127.10.98.3:46769)
0926 12:33:56.062521 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:56.062525 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:56.062544 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:33:56.062674 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:56.062678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:56.062734 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:56.062823 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:56.063263 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:56.063868 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:56.065033 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:56.065504 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:56.065586 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:56.065641 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:56.065892 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 12:33:56.066045 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:33:56.066223 18076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:56.062725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:56595 (local address 127.10.98.2:33289)
0926 12:33:56.062873 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:56.062877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:56.063054 (+ 177us) server_negotiation.cc:407] Connection header received
0926 12:33:56.063144 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:56.063147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:56.063201 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:56.063270 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:56.063499 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:56.064046 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:56.065211 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:56.065675 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:56.065781 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:56.065838 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:56.066030 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 12:33:56.066118 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 12:33:56.066290 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:56.062554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:33289 (local address 127.10.98.1:56595)
0926 12:33:56.062956 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:56.062968 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:56.063059 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:56.063287 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:56.063289 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:56.063306 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:56.063403 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:56.063409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:56.064178 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:56.064180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:56.065045 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:56.065055 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:56.065809 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:56.065812 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:56.065906 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:56.065928 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:33:56.066011 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 12:33:56.066079 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":17,"thread_start_us":108,"threads_started":1}
I20260926 12:33:56.066646 17928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d454455a9b154586b0ef7644e5941dd1" candidate_uuid: "d4f592a9801e467da0bc143fe5c41cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46015c6e40794d6486049f51816f2200" is_pre_election: true
I20260926 12:33:56.066733 17792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d454455a9b154586b0ef7644e5941dd1" candidate_uuid: "d4f592a9801e467da0bc143fe5c41cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08fc683fa8894fa685caba488359483d" is_pre_election: true
I20260926 12:33:56.066792 17928 raft_consensus.cc:2454] T d454455a9b154586b0ef7644e5941dd1 P 46015c6e40794d6486049f51816f2200 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4f592a9801e467da0bc143fe5c41cbc in term 0.
I20260926 12:33:56.066963 17792 raft_consensus.cc:2454] T d454455a9b154586b0ef7644e5941dd1 P 08fc683fa8894fa685caba488359483d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d4f592a9801e467da0bc143fe5c41cbc in term 0.
I20260926 12:33:56.067216 17589 leader_election.cc:302] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46015c6e40794d6486049f51816f2200, d4f592a9801e467da0bc143fe5c41cbc; no voters:
I20260926 12:33:56.067591 18082 raft_consensus.cc:2781] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:33:56.067653 18082 raft_consensus.cc:482] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:33:56.067682 18082 raft_consensus.cc:3037] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:56.068910 18082 raft_consensus.cc:504] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 } } peers { permanent_uuid: "d4f592a9801e467da0bc143fe5c41cbc" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45527 } } peers { permanent_uuid: "08fc683fa8894fa685caba488359483d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 33289 } }
I20260926 12:33:56.069314 18082 leader_election.cc:288] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc [CANDIDATE]: Term 1 election: Requested vote from peers 46015c6e40794d6486049f51816f2200 (127.10.98.3:46769), 08fc683fa8894fa685caba488359483d (127.10.98.2:33289)
I20260926 12:33:56.069653 17928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d454455a9b154586b0ef7644e5941dd1" candidate_uuid: "d4f592a9801e467da0bc143fe5c41cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46015c6e40794d6486049f51816f2200"
I20260926 12:33:56.069721 17792 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d454455a9b154586b0ef7644e5941dd1" candidate_uuid: "d4f592a9801e467da0bc143fe5c41cbc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08fc683fa8894fa685caba488359483d"
I20260926 12:33:56.069788 17928 raft_consensus.cc:3037] T d454455a9b154586b0ef7644e5941dd1 P 46015c6e40794d6486049f51816f2200 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:56.069857 17792 raft_consensus.cc:3037] T d454455a9b154586b0ef7644e5941dd1 P 08fc683fa8894fa685caba488359483d [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:56.070904 17792 raft_consensus.cc:2454] T d454455a9b154586b0ef7644e5941dd1 P 08fc683fa8894fa685caba488359483d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4f592a9801e467da0bc143fe5c41cbc in term 1.
I20260926 12:33:56.070905 17928 raft_consensus.cc:2454] T d454455a9b154586b0ef7644e5941dd1 P 46015c6e40794d6486049f51816f2200 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d4f592a9801e467da0bc143fe5c41cbc in term 1.
I20260926 12:33:56.071190 17589 leader_election.cc:302] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46015c6e40794d6486049f51816f2200, d4f592a9801e467da0bc143fe5c41cbc; no voters:
I20260926 12:33:56.071393 18082 raft_consensus.cc:2781] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:56.071633 18082 raft_consensus.cc:686] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc [term 1 LEADER]: Becoming Leader. State: Replica: d4f592a9801e467da0bc143fe5c41cbc, State: Running, Role: LEADER
I20260926 12:33:56.071905 18082 consensus_queue.cc:237] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 } } peers { permanent_uuid: "d4f592a9801e467da0bc143fe5c41cbc" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45527 } } peers { permanent_uuid: "08fc683fa8894fa685caba488359483d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 33289 } }
I20260926 12:33:56.072357 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:56.072043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.1:57227)
0926 12:33:56.072225 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:56.072283 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260926 12:33:56.074805 17338 catalog_manager.cc:5495] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc reported cstate change: term changed from 0 to 1, leader changed from <none> to d4f592a9801e467da0bc143fe5c41cbc (127.10.98.1). New cstate: current_term: 1 leader_uuid: "d4f592a9801e467da0bc143fe5c41cbc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d4f592a9801e467da0bc143fe5c41cbc" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "08fc683fa8894fa685caba488359483d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 33289 } health_report { overall_health: UNKNOWN } } }
I20260926 12:33:56.148664 18077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:56.144850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54660 (local address 127.10.98.1:45527)
0926 12:33:56.144992 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:56.144997 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:56.145011 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:33:56.145179 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:56.145182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:56.145247 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:56.145345 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:56.145578 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:56.146064 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:56.147225 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:56.147435 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:56.147522 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:56.147795 (+ 273us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:56.148273 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:56.148377 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:56.148462 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 12:33:56.148523 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:33:56.151751 17928 raft_consensus.cc:1261] T d454455a9b154586b0ef7644e5941dd1 P 46015c6e40794d6486049f51816f2200 [term 1 FOLLOWER]: Refusing update from remote peer d4f592a9801e467da0bc143fe5c41cbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:56.151926 17792 raft_consensus.cc:1261] T d454455a9b154586b0ef7644e5941dd1 P 08fc683fa8894fa685caba488359483d [term 1 FOLLOWER]: Refusing update from remote peer d4f592a9801e467da0bc143fe5c41cbc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:56.152254 18082 consensus_queue.cc:1035] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "46015c6e40794d6486049f51816f2200" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 46769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:56.152545 18088 consensus_queue.cc:1035] T d454455a9b154586b0ef7644e5941dd1 P d4f592a9801e467da0bc143fe5c41cbc [LEADER]: Connected to new peer: Peer: permanent_uuid: "08fc683fa8894fa685caba488359483d" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 33289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:56.154152 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:56.153846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.3:60227)
0926 12:33:56.154002 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:56.154066 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 12:33:56.154862 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:56.154309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.2:45483)
0926 12:33:56.154728 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:56.154785 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346}
I20260926 12:33:56.156955 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:56.156605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.1:43323)
0926 12:33:56.156820 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:56.156878 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260926 12:33:56.157321 18059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:56.156916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.3:39999)
0926 12:33:56.157201 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:56.157252 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220}
I20260926 12:33:56.161904 18087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:56.161508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.1:48391)
0926 12:33:56.161782 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:56.161838 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207}
I20260926 12:33:56.161972 18058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:56.161668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:40045 (local address 127.10.98.2:48831)
0926 12:33:56.161814 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:56.161865 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:40045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260926 12:33:56.166985 17338 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:56.167171 17338 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:56.171409 17338 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08b7010c899c498e9e6af4db7d5e95d2: Sending DeleteTablet for 3 replicas of tablet d454455a9b154586b0ef7644e5941dd1
I20260926 12:33:56.172175 17635 tablet_service.cc:1514] Processing DeleteTablet for tablet d454455a9b154586b0ef7644e5941dd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:56 UTC) from {username='oryx', principal='oryx/127.10.98.61@KRBTEST.COM'} at 127.0.0.1:54644
I20260926 12:33:56.172176 17772 tablet_service.cc:1514] Processing DeleteTablet for tablet d454455a9b154586b0ef7644e5941dd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:56 UTC) from {username='oryx', principal='oryx/127.10.98.61@KRBTEST.COM'} at 127.0.0.1:41464
I20260926 12:33:56.172523 17908 tablet_service.cc:1514] Processing DeleteTablet for tablet d454455a9b154586b0ef7644e5941dd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:56 UTC) from {username='oryx', principal='oryx/127.10.98.61@KRBTEST.COM'} at 127.0.0.1:40016
I20260926 12:33:56.172979 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 17483
I20260926 12:33:56.174911 18102 tablet_replica.cc:330] T d454455a9b154586b0ef7644e5941dd1 P 08fc683fa8894fa685caba488359483d: stopping tablet replica
I20260926 12:33:56.174940 18103 tablet_replica.cc:330] T d454455a9b154586b0ef7644e5941dd1 P 46015c6e40794d6486049f51816f2200: stopping tablet replica
I20260926 12:33:56.175048 18102 raft_consensus.cc:2229] T d454455a9b154586b0ef7644e5941dd1 P 08fc683fa8894fa685caba488359483d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:56.175061 18103 raft_consensus.cc:2229] T d454455a9b154586b0ef7644e5941dd1 P 46015c6e40794d6486049f51816f2200 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:56.175176 18102 raft_consensus.cc:2258] T d454455a9b154586b0ef7644e5941dd1 P 08fc683fa8894fa685caba488359483d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:56.175184 18103 raft_consensus.cc:2258] T d454455a9b154586b0ef7644e5941dd1 P 46015c6e40794d6486049f51816f2200 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:56.175870 18103 ts_tablet_manager.cc:1905] T d454455a9b154586b0ef7644e5941dd1 P 46015c6e40794d6486049f51816f2200: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:56.175935 18102 ts_tablet_manager.cc:1905] T d454455a9b154586b0ef7644e5941dd1 P 08fc683fa8894fa685caba488359483d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:56.178489 18103 ts_tablet_manager.cc:1918] T d454455a9b154586b0ef7644e5941dd1 P 46015c6e40794d6486049f51816f2200: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:33:56.178498 18102 ts_tablet_manager.cc:1918] T d454455a9b154586b0ef7644e5941dd1 P 08fc683fa8894fa685caba488359483d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:33:56.178574 18102 log.cc:1196] T d454455a9b154586b0ef7644e5941dd1 P 08fc683fa8894fa685caba488359483d: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/d454455a9b154586b0ef7644e5941dd1
I20260926 12:33:56.178574 18103 log.cc:1196] T d454455a9b154586b0ef7644e5941dd1 P 46015c6e40794d6486049f51816f2200: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/d454455a9b154586b0ef7644e5941dd1
I20260926 12:33:56.178828 18103 ts_tablet_manager.cc:1939] T d454455a9b154586b0ef7644e5941dd1 P 46015c6e40794d6486049f51816f2200: Deleting consensus metadata
I20260926 12:33:56.178829 18102 ts_tablet_manager.cc:1939] T d454455a9b154586b0ef7644e5941dd1 P 08fc683fa8894fa685caba488359483d: Deleting consensus metadata
I20260926 12:33:56.179680 17324 catalog_manager.cc:4841] TS 46015c6e40794d6486049f51816f2200 (127.10.98.3:46769): tablet d454455a9b154586b0ef7644e5941dd1 (table test-table [id=d4d7c77d630f4a32a33526f01713142f]) successfully deleted
I20260926 12:33:56.179865 17324 catalog_manager.cc:4841] TS 08fc683fa8894fa685caba488359483d (127.10.98.2:33289): tablet d454455a9b154586b0ef7644e5941dd1 (table test-table [id=d4d7c77d630f4a32a33526f01713142f]) successfully deleted
W20260926 12:33:56.180289 18075 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:56.179718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:45527 (local address 127.0.0.1:54668)
0926 12:33:56.179846 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:56.179861 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:56.179942 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:56.180180 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.1:45527: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 12:33:56.180444 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 17709
W20260926 12:33:56.180452 17323 catalog_manager.cc:4584] TS d4f592a9801e467da0bc143fe5c41cbc (127.10.98.1:45527): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d454455a9b154586b0ef7644e5941dd1: Network error: Client connection negotiation failed: client connection to 127.10.98.1:45527: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 12:33:56.186450 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 17845
I20260926 12:33:56.191346 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 17203
I20260926 12:33:56.196342 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 17311
2026-09-26T12:33:56Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7394 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:33:56 dist-test-slave-t7tk krb5kdc[18106](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:33:56 dist-test-slave-t7tk krb5kdc[18106](info): set up 2 sockets
Sep 26 12:33:56 dist-test-slave-t7tk krb5kdc[18106](info): commencing operation
krb5kdc: starting...
W20260926 12:33:57.162958 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:33:57 dist-test-slave-t7tk krb5kdc[18106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:33:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:33:57Z Disabled control of system clock
WARNING: no policy specified for oryx/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.62.keytab.
Entry for principal oryx/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.62.keytab.
I20260926 12:33:57.282063 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:37705
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.62.keytab
--principal=oryx/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:37705
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:57.392904 18119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:57.393213 18119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:57.393271 18119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:57.397279 18119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:33:57.397344 18119 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 12:33:57.397365 18119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:57.397388 18119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:57.397406 18119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:33:57.397428 18119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:33:57.402890 18119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:37705
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.62.keytab
--principal=oryx/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:37705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:57.403970 18119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:57.404996 18119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:57.413535 18127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:57.413558 18125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:57.413625 18119 server_base.cc:1034] running on GCE node
W20260926 12:33:57.413527 18124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:57.414412 18119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:57.415493 18119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:57.416710 18119 hybrid_clock.cc:648] HybridClock initialized: now 1790426037416689 us; error 34 us; skew 500 ppm
Sep 26 12:33:57 dist-test-slave-t7tk krb5kdc[18106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426037, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:57.419448 18119 init.cc:377] Logged in from keytab as oryx/127.10.98.62@KRBTEST.COM (short username oryx)
I20260926 12:33:57.420368 18119 webserver.cc:460] Webserver started at http://127.10.98.62:38089/ using document root <none> and password file <none>
I20260926 12:33:57.420931 18119 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:57.420991 18119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:57.421221 18119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:57.423239 18119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "1de7308f77074ee3872f1dd01d351299"
format_stamp: "Formatted at 2026-09-26 12:33:57 on dist-test-slave-t7tk"
server_key: "c8e8859adcb954cb89b00b26dbbb7eef"
server_key_iv: "727086abff5df07b1348ccade2f9a2fc"
server_key_version: "encryptionkey@0"
I20260926 12:33:57.423748 18119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "1de7308f77074ee3872f1dd01d351299"
format_stamp: "Formatted at 2026-09-26 12:33:57 on dist-test-slave-t7tk"
server_key: "c8e8859adcb954cb89b00b26dbbb7eef"
server_key_iv: "727086abff5df07b1348ccade2f9a2fc"
server_key_version: "encryptionkey@0"
I20260926 12:33:57.427466 18119 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:33:57.429952 18134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:57.431376 18119 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:33:57.431506 18119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "1de7308f77074ee3872f1dd01d351299"
format_stamp: "Formatted at 2026-09-26 12:33:57 on dist-test-slave-t7tk"
server_key: "c8e8859adcb954cb89b00b26dbbb7eef"
server_key_iv: "727086abff5df07b1348ccade2f9a2fc"
server_key_version: "encryptionkey@0"
I20260926 12:33:57.431607 18119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:57.467036 18119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:57.470350 18119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:57.470516 18119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:57.478269 18119 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:37705
I20260926 12:33:57.478288 18196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:37705 every 8 connection(s)
I20260926 12:33:57.479228 18119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:33:57.482007 18197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:57.487825 18197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299: Bootstrap starting.
I20260926 12:33:57.488756 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 18119
I20260926 12:33:57.488893 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:33:57.489240 10632 external_mini_cluster.cc:1428] Setting key e2c2afb0f6937ee1a39a210cf19154c5
I20260926 12:33:57.490840 18197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:57.491544 18197 log.cc:824] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:57.493785 18197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299: No bootstrap required, opened a new log
Sep 26 12:33:57 dist-test-slave-t7tk krb5kdc[18106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426037, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM
I20260926 12:33:57.498070 18197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de7308f77074ee3872f1dd01d351299" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 37705 } }
I20260926 12:33:57.498371 18197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:57.498430 18197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1de7308f77074ee3872f1dd01d351299, State: Initialized, Role: FOLLOWER
I20260926 12:33:57.498950 18197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de7308f77074ee3872f1dd01d351299" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 37705 } }
I20260926 12:33:57.499112 18197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:33:57.499177 18197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:33:57.499276 18197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:57.500229 18197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de7308f77074ee3872f1dd01d351299" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 37705 } }
I20260926 12:33:57.500554 18197 leader_election.cc:302] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1de7308f77074ee3872f1dd01d351299; no voters:
I20260926 12:33:57.500869 18197 leader_election.cc:288] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:33:57.501021 18202 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:57.501240 18202 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299 [term 1 LEADER]: Becoming Leader. State: Replica: 1de7308f77074ee3872f1dd01d351299, State: Running, Role: LEADER
I20260926 12:33:57.501545 18202 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de7308f77074ee3872f1dd01d351299" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 37705 } }
I20260926 12:33:57.502228 18197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:33:57.503305 18204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1de7308f77074ee3872f1dd01d351299. Latest consensus state: current_term: 1 leader_uuid: "1de7308f77074ee3872f1dd01d351299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de7308f77074ee3872f1dd01d351299" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 37705 } } }
I20260926 12:33:57.503424 18204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:57.503700 18203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1de7308f77074ee3872f1dd01d351299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1de7308f77074ee3872f1dd01d351299" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 37705 } } }
I20260926 12:33:57.503782 18203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299 [sys.catalog]: This master's current role is: LEADER
I20260926 12:33:57.503813 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:57.490429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46934 (local address 127.10.98.62:37705)
0926 12:33:57.490848 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:57.490866 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:57.490908 (+ 42us) server_negotiation.cc:407] Connection header received
0926 12:33:57.491574 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:57.491591 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:57.492025 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:57.492390 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:57.493250 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:57.494273 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:57.495394 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:57.495856 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:57.495965 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:57.498741 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:57.498748 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:57.498765 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:57.498797 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:57.500499 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:57.500995 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:57.501000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:57.501005 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:57.501209 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:57.501478 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:57.501481 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:57.501483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:57.501988 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:57.502185 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:57.502453 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 12:33:57.502769 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":130,"threads_started":1}
I20260926 12:33:57.504093 18211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:33:57.507043 18211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:33:57.512569 18211 catalog_manager.cc:1347] Generated new cluster ID: 76aa37b7ad6d4dac9332fe8d42829739
I20260926 12:33:57.512627 18211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:33:57.533756 18211 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:33:57.534765 18211 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:33:57.538954 18211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299: Generated new TSK 0
I20260926 12:33:57.539642 18211 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.10.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.1.keytab.
Entry for principal oryx/127.10.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.1.keytab.
I20260926 12:33:57.576386 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:37705
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.1.keytab
--principal=oryx/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:57.686707 18223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:57.686975 18223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:57.687021 18223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:57.691463 18223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:57.691542 18223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:57.691597 18223 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:33:57.691653 18223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:33:57.698979 18223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.1.keytab
--principal=oryx/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.98.62:37705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18223
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:57.700106 18223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:57.701177 18223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:57.710927 18228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:57.710924 18231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:57.710951 18229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:57.711295 18223 server_base.cc:1034] running on GCE node
I20260926 12:33:57.711722 18223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:57.712322 18223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:57.713471 18223 hybrid_clock.cc:648] HybridClock initialized: now 1790426037713458 us; error 26 us; skew 500 ppm
Sep 26 12:33:57 dist-test-slave-t7tk krb5kdc[18106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426037, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:57.716101 18223 init.cc:377] Logged in from keytab as oryx/127.10.98.1@KRBTEST.COM (short username oryx)
I20260926 12:33:57.716971 18223 webserver.cc:460] Webserver started at http://127.10.98.1:44901/ using document root <none> and password file <none>
I20260926 12:33:57.717549 18223 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:57.717619 18223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:57.717826 18223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:57.719830 18223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "b3c245430dbb4631a28158377c91d819"
format_stamp: "Formatted at 2026-09-26 12:33:57 on dist-test-slave-t7tk"
server_key: "70d8381c21d149f5b994f0e3eda09a56"
server_key_iv: "149d84ea5831c17033ef4912425e3f8e"
server_key_version: "encryptionkey@0"
I20260926 12:33:57.720353 18223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "b3c245430dbb4631a28158377c91d819"
format_stamp: "Formatted at 2026-09-26 12:33:57 on dist-test-slave-t7tk"
server_key: "70d8381c21d149f5b994f0e3eda09a56"
server_key_iv: "149d84ea5831c17033ef4912425e3f8e"
server_key_version: "encryptionkey@0"
I20260926 12:33:57.723968 18223 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 12:33:57.726464 18238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:57.727741 18223 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:33:57.727859 18223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "b3c245430dbb4631a28158377c91d819"
format_stamp: "Formatted at 2026-09-26 12:33:57 on dist-test-slave-t7tk"
server_key: "70d8381c21d149f5b994f0e3eda09a56"
server_key_iv: "149d84ea5831c17033ef4912425e3f8e"
server_key_version: "encryptionkey@0"
I20260926 12:33:57.727955 18223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:57.737244 18223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:57.740048 18223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:57.740221 18223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:57.741698 18223 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:57.741748 18223 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:57.741801 18223 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:57.741825 18223 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:33:57 dist-test-slave-t7tk krb5kdc[18106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426037, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM
I20260926 12:33:57.751892 18223 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:35743
I20260926 12:33:57.752792 18223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:33:57.753355 18353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:35743 every 8 connection(s)
I20260926 12:33:57.755066 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:57.742355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:60225 (local address 127.10.98.62:37705)
0926 12:33:57.742487 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:57.742491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:57.743335 (+ 844us) server_negotiation.cc:407] Connection header received
0926 12:33:57.744300 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:57.744304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:57.744363 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:57.744419 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:57.745854 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:57.746524 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:57.747856 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:57.748154 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:57.748258 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:57.751188 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:57.751191 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:57.751194 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:57.751221 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:57.752813 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:57.753452 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:57.753456 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:57.753458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:57.753550 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:57.753931 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:57.753934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:57.753936 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:57.754120 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:57.754207 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:57.754774 (+ 567us) server_negotiation.cc:299] Negotiation successful
0926 12:33:57.754906 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:33:57.755805 18247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:57.742720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:37705 (local address 127.10.98.1:60225)
0926 12:33:57.743175 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:57.743223 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:57.744050 (+ 827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:57.744814 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:57.744824 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:57.745205 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:57.745705 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:57.745725 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:57.746639 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:57.746642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:57.747714 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:57.747725 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:57.748330 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:57.748335 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:57.748551 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:57.749210 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:57.749230 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:57.750999 (+ 1769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:57.752938 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:57.752944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:57.752958 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:57.753329 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:57.753655 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:57.753658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:57.753660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:57.753821 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:57.754305 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:57.754310 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:57.754530 (+ 220us) client_negotiation.cc:769] Sending connection context
0926 12:33:57.754685 (+ 155us) client_negotiation.cc:240] Negotiation successful
0926 12:33:57.755010 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":141,"threads_started":1}
I20260926 12:33:57.757067 18354 heartbeater.cc:347] Connected to a master server at 127.10.98.62:37705
I20260926 12:33:57.757233 18354 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:57.757736 18354 heartbeater.cc:511] Master 127.10.98.62:37705 requested a full tablet report, sending...
I20260926 12:33:57.759276 18150 ts_manager.cc:194] Registered new tserver with Master: b3c245430dbb4631a28158377c91d819 (127.10.98.1:35743)
I20260926 12:33:57.760514 18150 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:60225
I20260926 12:33:57.762578 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 18223
I20260926 12:33:57.762698 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:33:57.762975 10632 external_mini_cluster.cc:1428] Setting key 5af212360bfb63df93bedac9c78ab07c
I20260926 12:33:57.765899 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:57.759112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:56239 (local address 127.10.98.62:37705)
0926 12:33:57.759225 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:57.759229 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:57.759361 (+ 132us) server_negotiation.cc:407] Connection header received
0926 12:33:57.759474 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:57.759477 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:57.759517 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:57.759597 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:57.760092 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:57.760593 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:57.761562 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:57.761774 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:57.761850 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:57.762979 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:57.762982 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:57.762984 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:57.763004 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:57.764408 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:57.764793 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:57.764797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:57.764799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:57.764871 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:57.765141 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:57.765145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:57.765147 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:57.765313 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:57.765397 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:57.765651 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 12:33:57.765761 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:33:57.765908 18360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:57.758999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:37705 (local address 127.10.98.1:56239)
0926 12:33:57.759279 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:57.759294 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:57.759392 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:57.759617 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:57.759620 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:57.759811 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:57.759922 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:57.759929 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:57.760751 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:57.760753 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:57.761454 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:57.761464 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:57.761879 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:57.761882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:57.761965 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:57.762436 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:57.762450 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:57.762857 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:57.764503 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:57.764506 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:57.764508 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:57.764691 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:57.764950 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:57.764954 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:57.764956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:57.765042 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:57.765400 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:57.765404 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:57.765492 (+ 88us) client_negotiation.cc:769] Sending connection context
0926 12:33:57.765579 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 12:33:57.765714 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":102,"threads_started":1}
WARNING: no policy specified for oryx/127.10.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.2.keytab.
Entry for principal oryx/127.10.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.2.keytab.
I20260926 12:33:57.788522 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:37705
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.2.keytab
--principal=oryx/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:57.897467 18363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:57.897783 18363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:57.897831 18363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:57.902036 18363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:57.902107 18363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:57.902153 18363 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:33:57.902206 18363 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:33:57.907872 18363 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.2.keytab
--principal=oryx/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.98.62:37705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18363
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:57.908954 18363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:57.909997 18363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:57.919634 18369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:57.919579 18371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:57.919584 18368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:57.919968 18363 server_base.cc:1034] running on GCE node
I20260926 12:33:57.920372 18363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:57.921010 18363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:57.922164 18363 hybrid_clock.cc:648] HybridClock initialized: now 1790426037922137 us; error 42 us; skew 500 ppm
Sep 26 12:33:57 dist-test-slave-t7tk krb5kdc[18106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426037, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:57.924968 18363 init.cc:377] Logged in from keytab as oryx/127.10.98.2@KRBTEST.COM (short username oryx)
I20260926 12:33:57.925997 18363 webserver.cc:460] Webserver started at http://127.10.98.2:45077/ using document root <none> and password file <none>
I20260926 12:33:57.926743 18363 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:57.926829 18363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:57.927037 18363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:57.929064 18363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "13f4dc3adf864c30a639b4240fd58dd3"
format_stamp: "Formatted at 2026-09-26 12:33:57 on dist-test-slave-t7tk"
server_key: "5b10cd00b7f4085bfe327ed1dd02b429"
server_key_iv: "ad3ffa8289dd9f7f182326cf550590aa"
server_key_version: "encryptionkey@0"
I20260926 12:33:57.929632 18363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "13f4dc3adf864c30a639b4240fd58dd3"
format_stamp: "Formatted at 2026-09-26 12:33:57 on dist-test-slave-t7tk"
server_key: "5b10cd00b7f4085bfe327ed1dd02b429"
server_key_iv: "ad3ffa8289dd9f7f182326cf550590aa"
server_key_version: "encryptionkey@0"
I20260926 12:33:57.933295 18363 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:33:57.935783 18378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:57.937078 18363 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:33:57.937288 18363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "13f4dc3adf864c30a639b4240fd58dd3"
format_stamp: "Formatted at 2026-09-26 12:33:57 on dist-test-slave-t7tk"
server_key: "5b10cd00b7f4085bfe327ed1dd02b429"
server_key_iv: "ad3ffa8289dd9f7f182326cf550590aa"
server_key_version: "encryptionkey@0"
I20260926 12:33:57.937391 18363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:57.958467 18363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:57.961814 18363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:57.961997 18363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:57.963516 18363 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:57.963568 18363 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:57.963622 18363 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:57.963651 18363 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:33:57 dist-test-slave-t7tk krb5kdc[18106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426037, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM
I20260926 12:33:57.973879 18363 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:40745
I20260926 12:33:57.973920 18493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:40745 every 8 connection(s)
I20260926 12:33:57.974881 18363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:33:57.976476 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:57.964355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:33821 (local address 127.10.98.62:37705)
0926 12:33:57.964496 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:57.964501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:57.965414 (+ 913us) server_negotiation.cc:407] Connection header received
0926 12:33:57.966434 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:57.966438 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:57.966496 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:57.966591 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:57.967830 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:57.968388 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:57.969589 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:57.969827 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:57.969924 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:57.972829 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:57.972832 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:57.972834 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:57.972860 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:57.974342 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:57.974898 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:57.974900 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:57.974901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:57.974986 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:57.975449 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:57.975453 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:57.975455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:57.975713 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:57.975814 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:57.976261 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 12:33:57.976361 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:33:57.977293 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:57.964765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:37705 (local address 127.10.98.2:33821)
0926 12:33:57.965250 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:57.965301 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:57.966225 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:57.966815 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:57.966823 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:57.967152 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:57.967691 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:57.967707 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:57.968507 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:57.968509 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:57.969477 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:57.969487 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:57.969971 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:57.969975 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:57.970162 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:57.970697 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:57.970718 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:57.972614 (+ 1896us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:57.974446 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:57.974451 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:57.974465 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:57.974811 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:57.975129 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:57.975132 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:57.975135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:57.975338 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:57.975818 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:57.975824 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:57.976067 (+ 243us) client_negotiation.cc:769] Sending connection context
0926 12:33:57.976223 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 12:33:57.976466 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":110,"threads_started":1}
I20260926 12:33:57.978531 18494 heartbeater.cc:347] Connected to a master server at 127.10.98.62:37705
I20260926 12:33:57.978670 18494 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:57.979137 18494 heartbeater.cc:511] Master 127.10.98.62:37705 requested a full tablet report, sending...
I20260926 12:33:57.980067 18151 ts_manager.cc:194] Registered new tserver with Master: 13f4dc3adf864c30a639b4240fd58dd3 (127.10.98.2:40745)
I20260926 12:33:57.980695 18151 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:33821
I20260926 12:33:57.984998 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 18363
I20260926 12:33:57.985133 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:33:57.985404 10632 external_mini_cluster.cc:1428] Setting key 713ae72a9dde2271d41854fbf7289e03
I20260926 12:33:57.987102 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:57.980493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:47821 (local address 127.10.98.62:37705)
0926 12:33:57.980608 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:57.980612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:57.980821 (+ 209us) server_negotiation.cc:407] Connection header received
0926 12:33:57.980920 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:57.980923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:57.980959 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:57.981028 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:57.981506 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:57.982006 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:57.982957 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:57.983179 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:57.983281 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:57.984166 (+ 885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:57.984169 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:57.984171 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:57.984192 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:57.985561 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:57.985947 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:57.985949 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:57.985950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:57.986021 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:57.986291 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:57.986293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:57.986294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:57.986504 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:57.986602 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:57.986888 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 12:33:57.986983 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:33:57.987118 18500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:57.980427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:37705 (local address 127.10.98.2:47821)
0926 12:33:57.980738 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:57.980750 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:57.980842 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:57.981061 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:57.981064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:57.981283 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:57.981399 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:57.981406 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:57.982122 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:57.982125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:57.982834 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:57.982842 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:57.983273 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:57.983275 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:57.983335 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:57.983754 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:57.983765 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:57.984047 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:57.985666 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:57.985670 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:57.985672 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:57.985857 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:57.986108 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:57.986111 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:57.986113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:57.986209 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:57.986597 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:57.986600 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:57.986689 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 12:33:57.986793 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 12:33:57.986931 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for oryx/127.10.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.3.keytab.
Entry for principal oryx/127.10.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.3.keytab.
I20260926 12:33:58.012341 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:37705
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.3.keytab
--principal=oryx/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:43293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:33:58.123073 18503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:33:58.123353 18503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:33:58.123399 18503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:33:58.127966 18503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:33:58.128036 18503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:33:58.128078 18503 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 12:33:58.128111 18503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:33:58.133863 18503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:43293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.3.keytab
--principal=oryx/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.10.98.62:37705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:33:58.135293 18503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:33:58.136690 18503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:33:58.148264 18509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:58.148439 18503 server_base.cc:1034] running on GCE node
W20260926 12:33:58.148264 18508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:33:58.148496 18511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:33:58.149008 18503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:33:58.149748 18503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:33:58.150923 18503 hybrid_clock.cc:648] HybridClock initialized: now 1790426038150898 us; error 37 us; skew 500 ppm
Sep 26 12:33:58 dist-test-slave-t7tk krb5kdc[18106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426038, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:33:58.153784 18503 init.cc:377] Logged in from keytab as oryx/127.10.98.3@KRBTEST.COM (short username oryx)
I20260926 12:33:58.154762 18503 webserver.cc:460] Webserver started at http://127.10.98.3:36863/ using document root <none> and password file <none>
I20260926 12:33:58.155305 18503 fs_manager.cc:359] Metadata directory not provided
I20260926 12:33:58.155370 18503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:33:58.155572 18503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:33:58.157748 18503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "d92e828418f149e4a847047d6fedc719"
format_stamp: "Formatted at 2026-09-26 12:33:58 on dist-test-slave-t7tk"
server_key: "4ad239434e69ddf7bd1d1f55020dce35"
server_key_iv: "72300586f47859d28556b3fe8aa368ce"
server_key_version: "encryptionkey@0"
I20260926 12:33:58.158330 18503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "d92e828418f149e4a847047d6fedc719"
format_stamp: "Formatted at 2026-09-26 12:33:58 on dist-test-slave-t7tk"
server_key: "4ad239434e69ddf7bd1d1f55020dce35"
server_key_iv: "72300586f47859d28556b3fe8aa368ce"
server_key_version: "encryptionkey@0"
I20260926 12:33:58.162302 18503 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:33:58.164942 18518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:58.166260 18503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:33:58.166375 18503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "d92e828418f149e4a847047d6fedc719"
format_stamp: "Formatted at 2026-09-26 12:33:58 on dist-test-slave-t7tk"
server_key: "4ad239434e69ddf7bd1d1f55020dce35"
server_key_iv: "72300586f47859d28556b3fe8aa368ce"
server_key_version: "encryptionkey@0"
I20260926 12:33:58.166473 18503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:33:58.180446 18503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:33:58.183528 18503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:33:58.183708 18503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:33:58.185381 18503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:33:58.185436 18503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:33:58.185491 18503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:33:58.185519 18503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 12:33:58 dist-test-slave-t7tk krb5kdc[18106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426038, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM
I20260926 12:33:58.195753 18503 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:38065
I20260926 12:33:58.195806 18633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:38065 every 8 connection(s)
I20260926 12:33:58.196653 18503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:33:58.198895 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 18503
I20260926 12:33:58.199023 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:33:58.199364 10632 external_mini_cluster.cc:1428] Setting key 60f813696443f7dd9737357f2827e41f
I20260926 12:33:58.200299 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.186160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:35249 (local address 127.10.98.62:37705)
0926 12:33:58.186335 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:58.186340 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:58.187225 (+ 885us) server_negotiation.cc:407] Connection header received
0926 12:33:58.188463 (+ 1238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:58.188467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:58.188547 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:58.188606 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:58.190480 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.191159 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.192833 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.193082 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.193196 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.196119 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:58.196129 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:58.196131 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:58.196158 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:58.197827 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:58.198363 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:58.198366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:58.198368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:58.198473 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:58.198997 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:58.199002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:58.199004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:58.199289 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:58.199374 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:58.200003 (+ 629us) server_negotiation.cc:299] Negotiation successful
0926 12:33:58.200131 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 12:33:58.201207 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.186567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:37705 (local address 127.10.98.3:35249)
0926 12:33:58.187041 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:58.187102 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:58.188185 (+ 1083us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:58.189055 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:58.189064 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:58.189579 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:58.190309 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.190331 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.191267 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.191271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.192715 (+ 1444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.192727 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.193248 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.193252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.193459 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.194173 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:58.194194 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:58.195938 (+ 1744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:58.197947 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:58.197953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:58.197964 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:58.198249 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:58.198633 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:58.198637 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:58.198647 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:58.198847 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:58.199403 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:58.199409 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:58.199757 (+ 348us) client_negotiation.cc:769] Sending connection context
0926 12:33:58.199932 (+ 175us) client_negotiation.cc:240] Negotiation successful
0926 12:33:58.200257 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":107,"threads_started":1}
I20260926 12:33:58.202485 18634 heartbeater.cc:347] Connected to a master server at 127.10.98.62:37705
I20260926 12:33:58.202689 18634 heartbeater.cc:464] Registering TS with master...
I20260926 12:33:58.203305 18634 heartbeater.cc:511] Master 127.10.98.62:37705 requested a full tablet report, sending...
I20260926 12:33:58.204502 18150 ts_manager.cc:194] Registered new tserver with Master: d92e828418f149e4a847047d6fedc719 (127.10.98.3:38065)
I20260926 12:33:58.205313 18150 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:35249
I20260926 12:33:58.211377 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.204769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:47659 (local address 127.10.98.62:37705)
0926 12:33:58.204886 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:58.204890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:58.205037 (+ 147us) server_negotiation.cc:407] Connection header received
0926 12:33:58.205145 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:58.205148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:58.205191 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:58.205272 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:58.205646 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.206229 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.207310 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.207531 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.207609 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.208590 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:58.208593 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:58.208595 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:58.208614 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:58.209929 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:58.210286 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:58.210289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:58.210291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:58.210364 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:58.210603 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:58.210607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:58.210609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:58.210786 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:58.210871 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:58.211114 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 12:33:58.211227 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:33:58.211386 18640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.204711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:37705 (local address 127.10.98.3:47659)
0926 12:33:58.204963 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:58.204977 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:58.205059 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:58.205284 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:58.205286 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:58.205447 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:58.205547 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.205553 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.206356 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.206359 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.207172 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.207182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.207631 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.207633 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.207704 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.208155 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:58.208168 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:58.208481 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:58.210005 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:58.210008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:58.210010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:58.210201 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:58.210429 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:58.210432 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:58.210433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:58.210498 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:58.210877 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:58.210880 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:58.210966 (+ 86us) client_negotiation.cc:769] Sending connection context
0926 12:33:58.211048 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 12:33:58.211187 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":1,"thread_start_us":98,"threads_started":1}
I20260926 12:33:58.213385 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 12:33:58 dist-test-slave-t7tk krb5kdc[18106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426038, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 12:33:58 dist-test-slave-t7tk krb5kdc[18106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:33:58 dist-test-slave-t7tk krb5kdc[18106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 12:33:58.236517 18200 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:58.230527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46940 (local address 127.10.98.62:37705)
0926 12:33:58.230650 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:58.230654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:58.230667 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:33:58.230709 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:58.230711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:58.230756 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:58.230831 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:58.231602 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.232135 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.232991 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.233273 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.233363 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.236361 (+ 2998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46940: BlockingRecv error: recv got EOF from 127.0.0.1:46940 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 12:33:58 dist-test-slave-t7tk krb5kdc[18106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426038, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM
I20260926 12:33:58.245640 18200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.237975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46942 (local address 127.10.98.62:37705)
0926 12:33:58.238093 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:58.238098 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:58.238230 (+ 132us) server_negotiation.cc:407] Connection header received
0926 12:33:58.238300 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:58.238302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:58.238344 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:58.238413 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:58.238934 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.239443 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.240202 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.240479 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.240588 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.242930 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:58.242933 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:58.242935 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:58.242959 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:58.244219 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:58.244617 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:58.244619 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:58.244621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:58.244717 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:58.244914 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:58.244916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:58.244918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:58.245079 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:58.245210 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:58.245430 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 12:33:58.245520 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:33:58.248574 18150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:33:58.251595 18150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:33:58.266276 18659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.260967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:40745 (local address 127.0.0.1:56120)
0926 12:33:58.261262 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:58.261274 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:58.261343 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:58.262558 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:58.262561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:58.262578 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:58.262665 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.262671 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.263879 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.263882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.264879 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.264888 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.265723 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.265727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.265833 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.265872 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 12:33:58.265982 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 12:33:58.266133 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"mutex_wait_us":23,"thread_start_us":109,"threads_started":1}
I20260926 12:33:58.266543 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.261730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56120 (local address 127.10.98.2:40745)
0926 12:33:58.262125 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:58.262136 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:58.262158 (+ 22us) server_negotiation.cc:407] Connection header received
0926 12:33:58.262229 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:58.262233 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:58.262425 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:58.262569 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:58.262771 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.263760 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.264994 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.265593 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.265703 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.265897 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:58.266008 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 12:33:58.266108 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":95,"threads_started":1}
I20260926 12:33:58.266621 18656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.260261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:38065 (local address 127.0.0.1:54422)
0926 12:33:58.260950 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:58.260998 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:58.261151 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:58.261932 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:58.261938 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:58.261963 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:58.262117 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.262125 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.263539 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.263542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.264903 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.264915 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.266194 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.266197 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.266294 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.266330 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 12:33:58.266397 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 12:33:58.266465 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":81,"spinlock_wait_cycles":4352,"thread_start_us":64,"threads_started":1}
I20260926 12:33:58.266773 18657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.260404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:35743 (local address 127.0.0.1:50814)
0926 12:33:58.260951 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:58.261000 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:58.261152 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:58.262179 (+ 1027us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:58.262182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:58.262200 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:58.262323 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.262330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.263350 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.263354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.264917 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.264924 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.266177 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.266180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.266321 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.266341 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 12:33:58.266593 (+ 252us) client_negotiation.cc:240] Negotiation successful
0926 12:33:58.266661 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":167,"spinlock_wait_cycles":5504,"thread_start_us":115,"threads_started":1}
I20260926 12:33:58.267133 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.261396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50814 (local address 127.10.98.1:35743)
0926 12:33:58.261752 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:58.261759 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:58.261781 (+ 22us) server_negotiation.cc:407] Connection header received
0926 12:33:58.261851 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:58.261856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:58.262052 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:58.262192 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:58.262429 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.263246 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.265043 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.265881 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.265968 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.266108 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:58.266921 (+ 813us) server_negotiation.cc:299] Negotiation successful
0926 12:33:58.267008 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":87,"threads_started":1}
I20260926 12:33:58.267151 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.260416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54422 (local address 127.10.98.3:38065)
0926 12:33:58.261409 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:58.261420 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:58.261444 (+ 24us) server_negotiation.cc:407] Connection header received
0926 12:33:58.261523 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:58.261540 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:58.261763 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:58.261923 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:58.262268 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.263427 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.265052 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.265868 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.265969 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.266146 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:58.266899 (+ 753us) server_negotiation.cc:299] Negotiation successful
0926 12:33:58.267004 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":889,"thread_start_us":124,"threads_started":1}
I20260926 12:33:58.269732 18428 tablet_service.cc:1467] Processing CreateTablet for tablet 41a48282cf964593ac3f498d89690564 (DEFAULT_TABLE table=test-table [id=7cb75d49817546e79c3e8e17b59e0aac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:58.270165 18568 tablet_service.cc:1467] Processing CreateTablet for tablet 41a48282cf964593ac3f498d89690564 (DEFAULT_TABLE table=test-table [id=7cb75d49817546e79c3e8e17b59e0aac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:58.270808 18428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a48282cf964593ac3f498d89690564. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:58.270694 18288 tablet_service.cc:1467] Processing CreateTablet for tablet 41a48282cf964593ac3f498d89690564 (DEFAULT_TABLE table=test-table [id=7cb75d49817546e79c3e8e17b59e0aac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:58.271216 18568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a48282cf964593ac3f498d89690564. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:58.271626 18288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a48282cf964593ac3f498d89690564. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:58.277153 18662 tablet_bootstrap.cc:492] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3: Bootstrap starting.
I20260926 12:33:58.279459 18662 tablet_bootstrap.cc:654] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:58.279459 18664 tablet_bootstrap.cc:492] T 41a48282cf964593ac3f498d89690564 P d92e828418f149e4a847047d6fedc719: Bootstrap starting.
I20260926 12:33:58.280171 18662 log.cc:824] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:58.280383 18663 tablet_bootstrap.cc:492] T 41a48282cf964593ac3f498d89690564 P b3c245430dbb4631a28158377c91d819: Bootstrap starting.
I20260926 12:33:58.281605 18664 tablet_bootstrap.cc:654] T 41a48282cf964593ac3f498d89690564 P d92e828418f149e4a847047d6fedc719: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:58.281980 18662 tablet_bootstrap.cc:492] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3: No bootstrap required, opened a new log
I20260926 12:33:58.282162 18662 ts_tablet_manager.cc:1397] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:33:58.282472 18663 tablet_bootstrap.cc:654] T 41a48282cf964593ac3f498d89690564 P b3c245430dbb4631a28158377c91d819: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:58.282493 18664 log.cc:824] T 41a48282cf964593ac3f498d89690564 P d92e828418f149e4a847047d6fedc719: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:58.283612 18663 log.cc:824] T 41a48282cf964593ac3f498d89690564 P b3c245430dbb4631a28158377c91d819: Log is configured to *not* fsync() on all Append() calls
I20260926 12:33:58.284521 18664 tablet_bootstrap.cc:492] T 41a48282cf964593ac3f498d89690564 P d92e828418f149e4a847047d6fedc719: No bootstrap required, opened a new log
I20260926 12:33:58.284716 18664 ts_tablet_manager.cc:1397] T 41a48282cf964593ac3f498d89690564 P d92e828418f149e4a847047d6fedc719: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:33:58.285467 18663 tablet_bootstrap.cc:492] T 41a48282cf964593ac3f498d89690564 P b3c245430dbb4631a28158377c91d819: No bootstrap required, opened a new log
I20260926 12:33:58.285481 18662 raft_consensus.cc:348] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } } peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } }
I20260926 12:33:58.285650 18663 ts_tablet_manager.cc:1397] T 41a48282cf964593ac3f498d89690564 P b3c245430dbb4631a28158377c91d819: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260926 12:33:58.285718 18662 raft_consensus.cc:374] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:58.285779 18662 raft_consensus.cc:729] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13f4dc3adf864c30a639b4240fd58dd3, State: Initialized, Role: FOLLOWER
I20260926 12:33:58.286187 18662 consensus_queue.cc:260] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } } peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } }
I20260926 12:33:58.286796 18494 heartbeater.cc:503] Master 127.10.98.62:37705 was elected leader, sending a full tablet report...
I20260926 12:33:58.288146 18662 ts_tablet_manager.cc:1428] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:33:58.289500 18664 raft_consensus.cc:348] T 41a48282cf964593ac3f498d89690564 P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } } peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } }
I20260926 12:33:58.289738 18664 raft_consensus.cc:374] T 41a48282cf964593ac3f498d89690564 P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:58.289789 18664 raft_consensus.cc:729] T 41a48282cf964593ac3f498d89690564 P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92e828418f149e4a847047d6fedc719, State: Initialized, Role: FOLLOWER
I20260926 12:33:58.289942 18663 raft_consensus.cc:348] T 41a48282cf964593ac3f498d89690564 P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } } peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } }
I20260926 12:33:58.290200 18664 consensus_queue.cc:260] T 41a48282cf964593ac3f498d89690564 P d92e828418f149e4a847047d6fedc719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } } peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } }
I20260926 12:33:58.290273 18663 raft_consensus.cc:374] T 41a48282cf964593ac3f498d89690564 P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:58.290329 18663 raft_consensus.cc:729] T 41a48282cf964593ac3f498d89690564 P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3c245430dbb4631a28158377c91d819, State: Initialized, Role: FOLLOWER
I20260926 12:33:58.290818 18634 heartbeater.cc:503] Master 127.10.98.62:37705 was elected leader, sending a full tablet report...
I20260926 12:33:58.291040 18663 consensus_queue.cc:260] T 41a48282cf964593ac3f498d89690564 P b3c245430dbb4631a28158377c91d819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } } peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } }
I20260926 12:33:58.291188 18664 ts_tablet_manager.cc:1428] T 41a48282cf964593ac3f498d89690564 P d92e828418f149e4a847047d6fedc719: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:33:58.291965 18663 ts_tablet_manager.cc:1428] T 41a48282cf964593ac3f498d89690564 P b3c245430dbb4631a28158377c91d819: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:33:58.291977 18354 heartbeater.cc:503] Master 127.10.98.62:37705 was elected leader, sending a full tablet report...
I20260926 12:33:58.383723 18668 raft_consensus.cc:482] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:33:58.383936 18668 raft_consensus.cc:504] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } } peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } }
I20260926 12:33:58.385133 18668 leader_election.cc:288] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d92e828418f149e4a847047d6fedc719 (127.10.98.3:38065), b3c245430dbb4631a28158377c91d819 (127.10.98.1:35743)
I20260926 12:33:58.389303 18671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.385398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:38065 (local address 127.10.98.2:43769)
0926 12:33:58.385861 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:58.385874 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:58.385965 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:58.386238 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:58.386241 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:58.386261 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:58.386378 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.386384 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.387147 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.387150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.388179 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.388188 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.388876 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.388880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.388976 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.389003 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 12:33:58.389071 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 12:33:58.389163 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":128,"thread_start_us":125,"threads_started":1}
I20260926 12:33:58.389346 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.385498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:43769 (local address 127.10.98.3:38065)
0926 12:33:58.385633 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:58.385637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:58.385954 (+ 317us) server_negotiation.cc:407] Connection header received
0926 12:33:58.386061 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:58.386064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:58.386115 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:58.386184 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:58.386479 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.387031 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.388307 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.388757 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.388834 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.388887 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:58.389138 (+ 251us) server_negotiation.cc:299] Negotiation successful
0926 12:33:58.389218 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:33:58.389526 18660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.385526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:38825 (local address 127.10.98.1:35743)
0926 12:33:58.385642 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:58.385646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:58.385672 (+ 26us) server_negotiation.cc:407] Connection header received
0926 12:33:58.385734 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:58.385737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:58.385787 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:58.385875 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:58.386234 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.386843 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.388478 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.388923 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.389028 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.389085 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:58.389344 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 12:33:58.389423 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:33:58.389503 18387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.385397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:35743 (local address 127.10.98.2:38825)
0926 12:33:58.385500 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:58.385515 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:58.385614 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:58.385889 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:58.385892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:58.385909 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:58.386047 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.386054 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.386957 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.386960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.388359 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.388369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.389080 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.389084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.389205 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.389226 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:33:58.389312 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 12:33:58.389381 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260926 12:33:58.390295 18578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a48282cf964593ac3f498d89690564" candidate_uuid: "13f4dc3adf864c30a639b4240fd58dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92e828418f149e4a847047d6fedc719" is_pre_election: true
I20260926 12:33:58.390295 18308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a48282cf964593ac3f498d89690564" candidate_uuid: "13f4dc3adf864c30a639b4240fd58dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3c245430dbb4631a28158377c91d819" is_pre_election: true
I20260926 12:33:58.390574 18578 raft_consensus.cc:2454] T 41a48282cf964593ac3f498d89690564 P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13f4dc3adf864c30a639b4240fd58dd3 in term 0.
I20260926 12:33:58.390575 18308 raft_consensus.cc:2454] T 41a48282cf964593ac3f498d89690564 P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13f4dc3adf864c30a639b4240fd58dd3 in term 0.
I20260926 12:33:58.391017 18382 leader_election.cc:302] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13f4dc3adf864c30a639b4240fd58dd3, d92e828418f149e4a847047d6fedc719; no voters:
I20260926 12:33:58.391275 18668 raft_consensus.cc:2781] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:33:58.391355 18668 raft_consensus.cc:482] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:33:58.391386 18668 raft_consensus.cc:3037] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:58.392414 18668 raft_consensus.cc:504] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } } peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } }
I20260926 12:33:58.392741 18668 leader_election.cc:288] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 [CANDIDATE]: Term 1 election: Requested vote from peers d92e828418f149e4a847047d6fedc719 (127.10.98.3:38065), b3c245430dbb4631a28158377c91d819 (127.10.98.1:35743)
I20260926 12:33:58.393074 18578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a48282cf964593ac3f498d89690564" candidate_uuid: "13f4dc3adf864c30a639b4240fd58dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92e828418f149e4a847047d6fedc719"
I20260926 12:33:58.393090 18308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a48282cf964593ac3f498d89690564" candidate_uuid: "13f4dc3adf864c30a639b4240fd58dd3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3c245430dbb4631a28158377c91d819"
I20260926 12:33:58.393215 18578 raft_consensus.cc:3037] T 41a48282cf964593ac3f498d89690564 P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:58.393275 18308 raft_consensus.cc:3037] T 41a48282cf964593ac3f498d89690564 P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:58.394356 18308 raft_consensus.cc:2454] T 41a48282cf964593ac3f498d89690564 P b3c245430dbb4631a28158377c91d819 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13f4dc3adf864c30a639b4240fd58dd3 in term 1.
I20260926 12:33:58.394387 18578 raft_consensus.cc:2454] T 41a48282cf964593ac3f498d89690564 P d92e828418f149e4a847047d6fedc719 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13f4dc3adf864c30a639b4240fd58dd3 in term 1.
I20260926 12:33:58.394645 18381 leader_election.cc:302] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13f4dc3adf864c30a639b4240fd58dd3, b3c245430dbb4631a28158377c91d819; no voters:
I20260926 12:33:58.394809 18668 raft_consensus.cc:2781] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:58.395004 18668 raft_consensus.cc:686] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 [term 1 LEADER]: Becoming Leader. State: Replica: 13f4dc3adf864c30a639b4240fd58dd3, State: Running, Role: LEADER
I20260926 12:33:58.395321 18668 consensus_queue.cc:237] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } } peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } }
I20260926 12:33:58.398501 18151 catalog_manager.cc:5495] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13f4dc3adf864c30a639b4240fd58dd3 (127.10.98.2). New cstate: current_term: 1 leader_uuid: "13f4dc3adf864c30a639b4240fd58dd3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } health_report { overall_health: HEALTHY } } }
I20260926 12:33:58.426411 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.422714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56124 (local address 127.10.98.2:40745)
0926 12:33:58.422830 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:58.422833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:58.422845 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:33:58.422918 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:58.422920 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:58.422972 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:58.423060 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:58.423363 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.423870 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.424823 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.425056 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.425253 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.425496 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:58.425944 (+ 448us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:58.426033 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:58.426244 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 12:33:58.426299 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:33:58.432430 18308 raft_consensus.cc:1261] T 41a48282cf964593ac3f498d89690564 P b3c245430dbb4631a28158377c91d819 [term 1 FOLLOWER]: Refusing update from remote peer 13f4dc3adf864c30a639b4240fd58dd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:58.432441 18578 raft_consensus.cc:1261] T 41a48282cf964593ac3f498d89690564 P d92e828418f149e4a847047d6fedc719 [term 1 FOLLOWER]: Refusing update from remote peer 13f4dc3adf864c30a639b4240fd58dd3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:58.433110 18672 consensus_queue.cc:1035] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:58.433298 18668 consensus_queue.cc:1035] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:58.442010 18677 mvcc.cc:205] Tried to move back new op lower bound from 7333585053411627008 to 7333585053269549056. Current Snapshot: MvccSnapshot[applied={T|T < 7333585053411627008}]
I20260926 12:33:58.448168 18150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:58.448321 18150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:58.450733 18150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299: Sending DeleteTablet for 3 replicas of tablet 41a48282cf964593ac3f498d89690564
I20260926 12:33:58.451674 18568 tablet_service.cc:1514] Processing DeleteTablet for tablet 41a48282cf964593ac3f498d89690564 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:58 UTC) from {username='oryx', principal='oryx/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:54422
I20260926 12:33:58.451866 18288 tablet_service.cc:1514] Processing DeleteTablet for tablet 41a48282cf964593ac3f498d89690564 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:58 UTC) from {username='oryx', principal='oryx/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:50814
I20260926 12:33:58.452373 18428 tablet_service.cc:1514] Processing DeleteTablet for tablet 41a48282cf964593ac3f498d89690564 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:58 UTC) from {username='oryx', principal='oryx/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:56120
I20260926 12:33:58.452818 18686 tablet_replica.cc:330] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3: stopping tablet replica
I20260926 12:33:58.453910 18687 tablet_replica.cc:330] T 41a48282cf964593ac3f498d89690564 P d92e828418f149e4a847047d6fedc719: stopping tablet replica
I20260926 12:33:58.453848 18150 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 12:33:58.454226 18687 raft_consensus.cc:2229] T 41a48282cf964593ac3f498d89690564 P d92e828418f149e4a847047d6fedc719 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:58.454546 18687 raft_consensus.cc:2258] T 41a48282cf964593ac3f498d89690564 P d92e828418f149e4a847047d6fedc719 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 12:33:58.454787 18150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:33:58.458057 18687 ts_tablet_manager.cc:1905] T 41a48282cf964593ac3f498d89690564 P d92e828418f149e4a847047d6fedc719: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:58.460960 18685 tablet_replica.cc:330] T 41a48282cf964593ac3f498d89690564 P b3c245430dbb4631a28158377c91d819: stopping tablet replica
I20260926 12:33:58.461141 18687 ts_tablet_manager.cc:1918] T 41a48282cf964593ac3f498d89690564 P d92e828418f149e4a847047d6fedc719: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:33:58.461313 18687 log.cc:1196] T 41a48282cf964593ac3f498d89690564 P d92e828418f149e4a847047d6fedc719: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/41a48282cf964593ac3f498d89690564
I20260926 12:33:58.461470 18685 raft_consensus.cc:2229] T 41a48282cf964593ac3f498d89690564 P b3c245430dbb4631a28158377c91d819 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:58.461773 18687 ts_tablet_manager.cc:1939] T 41a48282cf964593ac3f498d89690564 P d92e828418f149e4a847047d6fedc719: Deleting consensus metadata
I20260926 12:33:58.461833 18685 raft_consensus.cc:2258] T 41a48282cf964593ac3f498d89690564 P b3c245430dbb4631a28158377c91d819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:58.463248 18685 ts_tablet_manager.cc:1905] T 41a48282cf964593ac3f498d89690564 P b3c245430dbb4631a28158377c91d819: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:58.463233 18138 catalog_manager.cc:4841] TS d92e828418f149e4a847047d6fedc719 (127.10.98.3:38065): tablet 41a48282cf964593ac3f498d89690564 (table test-table [id=7cb75d49817546e79c3e8e17b59e0aac]) successfully deleted
I20260926 12:33:58.464027 18288 tablet_service.cc:1467] Processing CreateTablet for tablet 30806e83823f48c3a0f844209732ce29 (DEFAULT_TABLE table=test-table [id=86c87f6f753e4136aabc1fcdd5700a77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:58.464075 18428 tablet_service.cc:1467] Processing CreateTablet for tablet 30806e83823f48c3a0f844209732ce29 (DEFAULT_TABLE table=test-table [id=86c87f6f753e4136aabc1fcdd5700a77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:58.464270 18288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30806e83823f48c3a0f844209732ce29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:58.464352 18428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30806e83823f48c3a0f844209732ce29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:58.465714 18685 ts_tablet_manager.cc:1918] T 41a48282cf964593ac3f498d89690564 P b3c245430dbb4631a28158377c91d819: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:33:58.465806 18685 log.cc:1196] T 41a48282cf964593ac3f498d89690564 P b3c245430dbb4631a28158377c91d819: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-0/wal/wals/41a48282cf964593ac3f498d89690564
I20260926 12:33:58.466079 18685 ts_tablet_manager.cc:1939] T 41a48282cf964593ac3f498d89690564 P b3c245430dbb4631a28158377c91d819: Deleting consensus metadata
I20260926 12:33:58.466027 18568 tablet_service.cc:1467] Processing CreateTablet for tablet 30806e83823f48c3a0f844209732ce29 (DEFAULT_TABLE table=test-table [id=86c87f6f753e4136aabc1fcdd5700a77]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:58.466266 18568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 30806e83823f48c3a0f844209732ce29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:58.467589 18662 tablet_bootstrap.cc:492] T 30806e83823f48c3a0f844209732ce29 P 13f4dc3adf864c30a639b4240fd58dd3: Bootstrap starting.
I20260926 12:33:58.468452 18664 tablet_bootstrap.cc:492] T 30806e83823f48c3a0f844209732ce29 P d92e828418f149e4a847047d6fedc719: Bootstrap starting.
I20260926 12:33:58.468530 18686 raft_consensus.cc:2229] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:33:58.468549 18662 tablet_bootstrap.cc:654] T 30806e83823f48c3a0f844209732ce29 P 13f4dc3adf864c30a639b4240fd58dd3: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:58.468955 18686 raft_consensus.cc:2258] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:58.469381 18664 tablet_bootstrap.cc:654] T 30806e83823f48c3a0f844209732ce29 P d92e828418f149e4a847047d6fedc719: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:58.469535 18137 catalog_manager.cc:4841] TS b3c245430dbb4631a28158377c91d819 (127.10.98.1:35743): tablet 41a48282cf964593ac3f498d89690564 (table test-table [id=7cb75d49817546e79c3e8e17b59e0aac]) successfully deleted
I20260926 12:33:58.469898 18662 tablet_bootstrap.cc:492] T 30806e83823f48c3a0f844209732ce29 P 13f4dc3adf864c30a639b4240fd58dd3: No bootstrap required, opened a new log
I20260926 12:33:58.469976 18662 ts_tablet_manager.cc:1397] T 30806e83823f48c3a0f844209732ce29 P 13f4dc3adf864c30a639b4240fd58dd3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260926 12:33:58.470441 18662 raft_consensus.cc:348] T 30806e83823f48c3a0f844209732ce29 P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:58.470547 18662 raft_consensus.cc:374] T 30806e83823f48c3a0f844209732ce29 P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:58.470577 18662 raft_consensus.cc:729] T 30806e83823f48c3a0f844209732ce29 P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13f4dc3adf864c30a639b4240fd58dd3, State: Initialized, Role: FOLLOWER
I20260926 12:33:58.470625 18664 tablet_bootstrap.cc:492] T 30806e83823f48c3a0f844209732ce29 P d92e828418f149e4a847047d6fedc719: No bootstrap required, opened a new log
I20260926 12:33:58.470700 18664 ts_tablet_manager.cc:1397] T 30806e83823f48c3a0f844209732ce29 P d92e828418f149e4a847047d6fedc719: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 12:33:58.470690 18662 consensus_queue.cc:260] T 30806e83823f48c3a0f844209732ce29 P 13f4dc3adf864c30a639b4240fd58dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:58.470933 18662 ts_tablet_manager.cc:1428] T 30806e83823f48c3a0f844209732ce29 P 13f4dc3adf864c30a639b4240fd58dd3: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 12:33:58.471100 18664 raft_consensus.cc:348] T 30806e83823f48c3a0f844209732ce29 P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:58.471195 18664 raft_consensus.cc:374] T 30806e83823f48c3a0f844209732ce29 P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:58.471216 18664 raft_consensus.cc:729] T 30806e83823f48c3a0f844209732ce29 P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92e828418f149e4a847047d6fedc719, State: Initialized, Role: FOLLOWER
I20260926 12:33:58.471210 18663 tablet_bootstrap.cc:492] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819: Bootstrap starting.
I20260926 12:33:58.471315 18664 consensus_queue.cc:260] T 30806e83823f48c3a0f844209732ce29 P d92e828418f149e4a847047d6fedc719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:58.471540 18664 ts_tablet_manager.cc:1428] T 30806e83823f48c3a0f844209732ce29 P d92e828418f149e4a847047d6fedc719: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:58.472110 18663 tablet_bootstrap.cc:654] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:58.474356 18663 tablet_bootstrap.cc:492] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819: No bootstrap required, opened a new log
I20260926 12:33:58.474442 18663 ts_tablet_manager.cc:1397] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 12:33:58.475153 18663 raft_consensus.cc:348] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:58.475268 18663 raft_consensus.cc:374] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:58.475306 18663 raft_consensus.cc:729] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3c245430dbb4631a28158377c91d819, State: Initialized, Role: FOLLOWER
I20260926 12:33:58.475416 18663 consensus_queue.cc:260] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:58.475652 18663 ts_tablet_manager.cc:1428] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20260926 12:33:58.475999 18495 tablet.cc:2369] T 30806e83823f48c3a0f844209732ce29 P 13f4dc3adf864c30a639b4240fd58dd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:33:58.477679 18686 ts_tablet_manager.cc:1905] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:58.480311 18686 ts_tablet_manager.cc:1918] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:33:58.480405 18686 log.cc:1196] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/41a48282cf964593ac3f498d89690564
I20260926 12:33:58.480687 18686 ts_tablet_manager.cc:1939] T 41a48282cf964593ac3f498d89690564 P 13f4dc3adf864c30a639b4240fd58dd3: Deleting consensus metadata
I20260926 12:33:58.481868 18137 catalog_manager.cc:4841] TS 13f4dc3adf864c30a639b4240fd58dd3 (127.10.98.2:40745): tablet 41a48282cf964593ac3f498d89690564 (table test-table [id=7cb75d49817546e79c3e8e17b59e0aac]) successfully deleted
W20260926 12:33:58.504585 18355 tablet.cc:2369] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:33:58.698395 18635 tablet.cc:2369] T 30806e83823f48c3a0f844209732ce29 P d92e828418f149e4a847047d6fedc719: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:33:58.724396 18670 raft_consensus.cc:482] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:33:58.724568 18670 raft_consensus.cc:504] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:58.725788 18670 leader_election.cc:288] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13f4dc3adf864c30a639b4240fd58dd3 (127.10.98.2:40745), d92e828418f149e4a847047d6fedc719 (127.10.98.3:38065)
I20260926 12:33:58.730388 18658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.726145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:44281 (local address 127.10.98.3:38065)
0926 12:33:58.726286 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:58.726291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:58.726440 (+ 149us) server_negotiation.cc:407] Connection header received
0926 12:33:58.726637 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:58.726640 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:58.726685 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:58.726721 (+ 36us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:58.727055 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.727576 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.729219 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.729711 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.729822 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.729889 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:58.730192 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 12:33:58.730279 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 12:33:58.730422 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.726053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:38065 (local address 127.10.98.1:44281)
0926 12:33:58.726349 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:58.726362 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:58.726497 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:58.726845 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:58.726847 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:58.726859 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:58.726952 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.726957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.727707 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.727711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.728922 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.728934 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.729867 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.729870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.729956 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.729985 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 12:33:58.730158 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 12:33:58.730252 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":20,"thread_start_us":70,"threads_started":1}
I20260926 12:33:58.730600 18692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.726134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:40745 (local address 127.10.98.1:52917)
0926 12:33:58.726451 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:58.726462 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:58.726554 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:58.726791 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:58.726794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:58.726813 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:58.726946 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.726953 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.728276 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.728279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.729384 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.729394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.730211 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.730215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.730307 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.730328 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:33:58.730398 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 12:33:58.730458 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":34,"thread_start_us":99,"threads_started":1}
I20260926 12:33:58.730613 18661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.726243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:52917 (local address 127.10.98.2:40745)
0926 12:33:58.726366 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:58.726370 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:58.726528 (+ 158us) server_negotiation.cc:407] Connection header received
0926 12:33:58.726646 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:58.726649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:58.726699 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:58.726765 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:58.727699 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.728160 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.729498 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.730094 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.730183 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.730242 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:58.730422 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 12:33:58.730496 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:33:58.730934 18578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30806e83823f48c3a0f844209732ce29" candidate_uuid: "b3c245430dbb4631a28158377c91d819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92e828418f149e4a847047d6fedc719" is_pre_election: true
I20260926 12:33:58.731096 18578 raft_consensus.cc:2454] T 30806e83823f48c3a0f844209732ce29 P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3c245430dbb4631a28158377c91d819 in term 0.
I20260926 12:33:58.731185 18448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30806e83823f48c3a0f844209732ce29" candidate_uuid: "b3c245430dbb4631a28158377c91d819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f4dc3adf864c30a639b4240fd58dd3" is_pre_election: true
I20260926 12:33:58.731421 18448 raft_consensus.cc:2454] T 30806e83823f48c3a0f844209732ce29 P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3c245430dbb4631a28158377c91d819 in term 0.
I20260926 12:33:58.731436 18242 leader_election.cc:302] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3c245430dbb4631a28158377c91d819, d92e828418f149e4a847047d6fedc719; no voters:
I20260926 12:33:58.731742 18670 raft_consensus.cc:2781] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:33:58.731802 18670 raft_consensus.cc:482] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:33:58.731842 18670 raft_consensus.cc:3037] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:58.732790 18670 raft_consensus.cc:504] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:58.733168 18670 leader_election.cc:288] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 [CANDIDATE]: Term 1 election: Requested vote from peers 13f4dc3adf864c30a639b4240fd58dd3 (127.10.98.2:40745), d92e828418f149e4a847047d6fedc719 (127.10.98.3:38065)
I20260926 12:33:58.733448 18448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30806e83823f48c3a0f844209732ce29" candidate_uuid: "b3c245430dbb4631a28158377c91d819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f4dc3adf864c30a639b4240fd58dd3"
I20260926 12:33:58.733486 18578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "30806e83823f48c3a0f844209732ce29" candidate_uuid: "b3c245430dbb4631a28158377c91d819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92e828418f149e4a847047d6fedc719"
I20260926 12:33:58.733553 18448 raft_consensus.cc:3037] T 30806e83823f48c3a0f844209732ce29 P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:58.733608 18578 raft_consensus.cc:3037] T 30806e83823f48c3a0f844209732ce29 P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:58.734488 18448 raft_consensus.cc:2454] T 30806e83823f48c3a0f844209732ce29 P 13f4dc3adf864c30a639b4240fd58dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3c245430dbb4631a28158377c91d819 in term 1.
I20260926 12:33:58.734628 18578 raft_consensus.cc:2454] T 30806e83823f48c3a0f844209732ce29 P d92e828418f149e4a847047d6fedc719 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3c245430dbb4631a28158377c91d819 in term 1.
I20260926 12:33:58.734778 18241 leader_election.cc:302] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13f4dc3adf864c30a639b4240fd58dd3, b3c245430dbb4631a28158377c91d819; no voters:
I20260926 12:33:58.734933 18670 raft_consensus.cc:2781] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:58.735132 18670 raft_consensus.cc:686] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 [term 1 LEADER]: Becoming Leader. State: Replica: b3c245430dbb4631a28158377c91d819, State: Running, Role: LEADER
I20260926 12:33:58.735383 18670 consensus_queue.cc:237] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:58.737550 18151 catalog_manager.cc:5495] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3c245430dbb4631a28158377c91d819 (127.10.98.1). New cstate: current_term: 1 leader_uuid: "b3c245430dbb4631a28158377c91d819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } health_report { overall_health: UNKNOWN } } }
Sep 26 12:33:58 dist-test-slave-t7tk krb5kdc[18106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426037, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.62@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM
I20260926 12:33:58.795114 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.786848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:37705 (local address 127.0.0.1:41484)
0926 12:33:58.787146 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:58.787162 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:58.787291 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:58.787534 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:58.787537 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:58.787995 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:33:58.788098 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.788104 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.788835 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.788838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.789443 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.789452 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.789877 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.789880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.789951 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.790500 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:33:58.790513 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:33:58.791947 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:33:58.793598 (+ 1651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:58.793609 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:58.793616 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:58.793851 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:58.794090 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:33:58.794093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:33:58.794094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:33:58.794165 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:33:58.794531 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:33:58.794538 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:33:58.794644 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 12:33:58.794749 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 12:33:58.794913 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":85,"threads_started":1}
I20260926 12:33:58.795113 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.786912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41484 (local address 127.10.98.62:37705)
0926 12:33:58.787168 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:58.787172 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:58.787226 (+ 54us) server_negotiation.cc:407] Connection header received
0926 12:33:58.787384 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:58.787387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:58.787425 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:58.787498 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:58.788209 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.788736 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.789564 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.789780 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.789865 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.792073 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:58.792076 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:58.792078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:58.792099 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:58.793502 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:58.793939 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:58.793943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:58.793944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:58.794012 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:58.794251 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:58.794254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:58.794256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:58.794431 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:58.794515 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:58.794798 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 12:33:58.794970 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":205,"thread_start_us":68,"threads_started":1}
I20260926 12:33:58.799129 18151 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:41484:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 12:33:58.799904 18151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:33:58.804906 18288 tablet_service.cc:1467] Processing CreateTablet for tablet fd733d18b51e447a8dc0b20cd0c0408b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ce2fc48f26cb4ec89c8041799b6443da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:33:58.804947 18428 tablet_service.cc:1467] Processing CreateTablet for tablet fd733d18b51e447a8dc0b20cd0c0408b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ce2fc48f26cb4ec89c8041799b6443da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:33:58.805161 18568 tablet_service.cc:1467] Processing CreateTablet for tablet fd733d18b51e447a8dc0b20cd0c0408b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ce2fc48f26cb4ec89c8041799b6443da]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 12:33:58.805347 18428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd733d18b51e447a8dc0b20cd0c0408b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:58.805347 18288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd733d18b51e447a8dc0b20cd0c0408b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:58.805553 18568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fd733d18b51e447a8dc0b20cd0c0408b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:58.808069 18664 tablet_bootstrap.cc:492] T fd733d18b51e447a8dc0b20cd0c0408b P d92e828418f149e4a847047d6fedc719: Bootstrap starting.
I20260926 12:33:58.808120 18663 tablet_bootstrap.cc:492] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819: Bootstrap starting.
I20260926 12:33:58.808841 18663 tablet_bootstrap.cc:654] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:58.808857 18664 tablet_bootstrap.cc:654] T fd733d18b51e447a8dc0b20cd0c0408b P d92e828418f149e4a847047d6fedc719: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:58.809453 18662 tablet_bootstrap.cc:492] T fd733d18b51e447a8dc0b20cd0c0408b P 13f4dc3adf864c30a639b4240fd58dd3: Bootstrap starting.
I20260926 12:33:58.810110 18663 tablet_bootstrap.cc:492] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819: No bootstrap required, opened a new log
I20260926 12:33:58.810201 18663 ts_tablet_manager.cc:1397] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 12:33:58.810299 18662 tablet_bootstrap.cc:654] T fd733d18b51e447a8dc0b20cd0c0408b P 13f4dc3adf864c30a639b4240fd58dd3: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:58.810645 18663 raft_consensus.cc:348] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:58.810758 18663 raft_consensus.cc:374] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:58.810789 18663 raft_consensus.cc:729] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3c245430dbb4631a28158377c91d819, State: Initialized, Role: FOLLOWER
I20260926 12:33:58.810916 18663 consensus_queue.cc:260] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:58.811081 18664 tablet_bootstrap.cc:492] T fd733d18b51e447a8dc0b20cd0c0408b P d92e828418f149e4a847047d6fedc719: No bootstrap required, opened a new log
I20260926 12:33:58.811169 18664 ts_tablet_manager.cc:1397] T fd733d18b51e447a8dc0b20cd0c0408b P d92e828418f149e4a847047d6fedc719: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 12:33:58.811203 18663 ts_tablet_manager.cc:1428] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:58.811360 18662 tablet_bootstrap.cc:492] T fd733d18b51e447a8dc0b20cd0c0408b P 13f4dc3adf864c30a639b4240fd58dd3: No bootstrap required, opened a new log
I20260926 12:33:58.811436 18662 ts_tablet_manager.cc:1397] T fd733d18b51e447a8dc0b20cd0c0408b P 13f4dc3adf864c30a639b4240fd58dd3: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 12:33:58.811390 18670 tablet_replica.cc:439] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } } }
I20260926 12:33:58.811525 18670 tablet_replica.cc:442] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:33:58.811589 18664 raft_consensus.cc:348] T fd733d18b51e447a8dc0b20cd0c0408b P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:58.811707 18664 raft_consensus.cc:374] T fd733d18b51e447a8dc0b20cd0c0408b P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:58.811739 18664 raft_consensus.cc:729] T fd733d18b51e447a8dc0b20cd0c0408b P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92e828418f149e4a847047d6fedc719, State: Initialized, Role: FOLLOWER
I20260926 12:33:58.811800 18662 raft_consensus.cc:348] T fd733d18b51e447a8dc0b20cd0c0408b P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:58.811892 18662 raft_consensus.cc:374] T fd733d18b51e447a8dc0b20cd0c0408b P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:58.811857 18664 consensus_queue.cc:260] T fd733d18b51e447a8dc0b20cd0c0408b P d92e828418f149e4a847047d6fedc719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:58.811926 18662 raft_consensus.cc:729] T fd733d18b51e447a8dc0b20cd0c0408b P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13f4dc3adf864c30a639b4240fd58dd3, State: Initialized, Role: FOLLOWER
I20260926 12:33:58.812029 18662 consensus_queue.cc:260] T fd733d18b51e447a8dc0b20cd0c0408b P 13f4dc3adf864c30a639b4240fd58dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:58.812112 18664 ts_tablet_manager.cc:1428] T fd733d18b51e447a8dc0b20cd0c0408b P d92e828418f149e4a847047d6fedc719: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 12:33:58.812286 18662 ts_tablet_manager.cc:1428] T fd733d18b51e447a8dc0b20cd0c0408b P 13f4dc3adf864c30a639b4240fd58dd3: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260926 12:33:58.812319 18669 tablet_replica.cc:439] T fd733d18b51e447a8dc0b20cd0c0408b P d92e828418f149e4a847047d6fedc719: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } } }
I20260926 12:33:58.812459 18669 tablet_replica.cc:442] T fd733d18b51e447a8dc0b20cd0c0408b P d92e828418f149e4a847047d6fedc719: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:33:58.812474 18681 tablet_replica.cc:439] T fd733d18b51e447a8dc0b20cd0c0408b P 13f4dc3adf864c30a639b4240fd58dd3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } } }
I20260926 12:33:58.812587 18681 tablet_replica.cc:442] T fd733d18b51e447a8dc0b20cd0c0408b P 13f4dc3adf864c30a639b4240fd58dd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:33:58.884444 18670 raft_consensus.cc:482] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:33:58.884595 18670 raft_consensus.cc:504] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:58.884999 18670 leader_election.cc:288] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13f4dc3adf864c30a639b4240fd58dd3 (127.10.98.2:40745), d92e828418f149e4a847047d6fedc719 (127.10.98.3:38065)
I20260926 12:33:58.885483 18448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd733d18b51e447a8dc0b20cd0c0408b" candidate_uuid: "b3c245430dbb4631a28158377c91d819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f4dc3adf864c30a639b4240fd58dd3" is_pre_election: true
I20260926 12:33:58.885492 18578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd733d18b51e447a8dc0b20cd0c0408b" candidate_uuid: "b3c245430dbb4631a28158377c91d819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92e828418f149e4a847047d6fedc719" is_pre_election: true
I20260926 12:33:58.885635 18578 raft_consensus.cc:2454] T fd733d18b51e447a8dc0b20cd0c0408b P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3c245430dbb4631a28158377c91d819 in term 0.
I20260926 12:33:58.885641 18448 raft_consensus.cc:2454] T fd733d18b51e447a8dc0b20cd0c0408b P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3c245430dbb4631a28158377c91d819 in term 0.
I20260926 12:33:58.885934 18242 leader_election.cc:302] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3c245430dbb4631a28158377c91d819, d92e828418f149e4a847047d6fedc719; no voters:
I20260926 12:33:58.886199 18670 raft_consensus.cc:2781] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:33:58.886256 18670 raft_consensus.cc:482] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:33:58.886303 18670 raft_consensus.cc:3037] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:58.887254 18670 raft_consensus.cc:504] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:58.887552 18670 leader_election.cc:288] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819 [CANDIDATE]: Term 1 election: Requested vote from peers 13f4dc3adf864c30a639b4240fd58dd3 (127.10.98.2:40745), d92e828418f149e4a847047d6fedc719 (127.10.98.3:38065)
I20260926 12:33:58.887900 18448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd733d18b51e447a8dc0b20cd0c0408b" candidate_uuid: "b3c245430dbb4631a28158377c91d819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f4dc3adf864c30a639b4240fd58dd3"
I20260926 12:33:58.887900 18578 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fd733d18b51e447a8dc0b20cd0c0408b" candidate_uuid: "b3c245430dbb4631a28158377c91d819" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d92e828418f149e4a847047d6fedc719"
I20260926 12:33:58.888020 18578 raft_consensus.cc:3037] T fd733d18b51e447a8dc0b20cd0c0408b P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:58.888020 18448 raft_consensus.cc:3037] T fd733d18b51e447a8dc0b20cd0c0408b P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:58.889089 18448 raft_consensus.cc:2454] T fd733d18b51e447a8dc0b20cd0c0408b P 13f4dc3adf864c30a639b4240fd58dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3c245430dbb4631a28158377c91d819 in term 1.
I20260926 12:33:58.889133 18578 raft_consensus.cc:2454] T fd733d18b51e447a8dc0b20cd0c0408b P d92e828418f149e4a847047d6fedc719 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3c245430dbb4631a28158377c91d819 in term 1.
I20260926 12:33:58.889380 18242 leader_election.cc:302] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3c245430dbb4631a28158377c91d819, d92e828418f149e4a847047d6fedc719; no voters:
I20260926 12:33:58.889536 18670 raft_consensus.cc:2781] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:58.889624 18670 raft_consensus.cc:686] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819 [term 1 LEADER]: Becoming Leader. State: Replica: b3c245430dbb4631a28158377c91d819, State: Running, Role: LEADER
I20260926 12:33:58.889760 18670 consensus_queue.cc:237] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:58.890386 18693 tablet_replica.cc:439] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819: TxnStatusTablet state changed. Reason: New leader b3c245430dbb4631a28158377c91d819. Latest consensus state: current_term: 1 leader_uuid: "b3c245430dbb4631a28158377c91d819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } } }
I20260926 12:33:58.890486 18693 tablet_replica.cc:442] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:33:58.890753 18706 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 12:33:58.890949 18706 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 12:33:58.891484 18150 catalog_manager.cc:5495] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819 reported cstate change: term changed from 0 to 1, leader changed from <none> to b3c245430dbb4631a28158377c91d819 (127.10.98.1). New cstate: current_term: 1 leader_uuid: "b3c245430dbb4631a28158377c91d819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } health_report { overall_health: UNKNOWN } } }
I20260926 12:33:58.911514 18700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.907599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:35743 (local address 127.0.0.1:52930)
0926 12:33:58.907817 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:58.907831 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:58.907947 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:58.908312 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:58.908315 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:58.908326 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:33:58.908418 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.908422 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.909338 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.909341 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.909950 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.909958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.910401 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.910405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.910495 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.910538 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:33:58.911198 (+ 660us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:33:58.911222 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:33:58.911287 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 12:33:58.911380 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260926 12:33:58.911531 18709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.907710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52930 (local address 127.10.98.1:35743)
0926 12:33:58.907987 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:58.907995 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:58.908010 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:33:58.908081 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:58.908084 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:58.908150 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:58.908246 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:58.908529 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.909237 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.910065 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.910305 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.910391 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.910678 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:58.911108 (+ 430us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:58.911212 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:58.911312 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 12:33:58.911370 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":93,"threads_started":1}
I20260926 12:33:58.914624 18578 raft_consensus.cc:1261] T fd733d18b51e447a8dc0b20cd0c0408b P d92e828418f149e4a847047d6fedc719 [term 1 FOLLOWER]: Refusing update from remote peer b3c245430dbb4631a28158377c91d819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:58.914940 18448 raft_consensus.cc:1261] T fd733d18b51e447a8dc0b20cd0c0408b P 13f4dc3adf864c30a639b4240fd58dd3 [term 1 FOLLOWER]: Refusing update from remote peer b3c245430dbb4631a28158377c91d819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:58.915227 18670 consensus_queue.cc:1035] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:58.915570 18693 consensus_queue.cc:1035] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:58.916777 18669 tablet_replica.cc:439] T fd733d18b51e447a8dc0b20cd0c0408b P d92e828418f149e4a847047d6fedc719: TxnStatusTablet state changed. Reason: New leader b3c245430dbb4631a28158377c91d819. Latest consensus state: current_term: 1 leader_uuid: "b3c245430dbb4631a28158377c91d819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } } }
I20260926 12:33:58.916896 18669 tablet_replica.cc:442] T fd733d18b51e447a8dc0b20cd0c0408b P d92e828418f149e4a847047d6fedc719: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:33:58.917246 18681 tablet_replica.cc:439] T fd733d18b51e447a8dc0b20cd0c0408b P 13f4dc3adf864c30a639b4240fd58dd3: TxnStatusTablet state changed. Reason: New leader b3c245430dbb4631a28158377c91d819. Latest consensus state: current_term: 1 leader_uuid: "b3c245430dbb4631a28158377c91d819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } } }
I20260926 12:33:58.917363 18681 tablet_replica.cc:442] T fd733d18b51e447a8dc0b20cd0c0408b P 13f4dc3adf864c30a639b4240fd58dd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:33:58.918524 18670 tablet_replica.cc:439] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3c245430dbb4631a28158377c91d819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } } }
I20260926 12:33:58.918632 18670 tablet_replica.cc:442] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:33:58.919123 18693 tablet_replica.cc:439] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b3c245430dbb4631a28158377c91d819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } } }
I20260926 12:33:58.919209 18693 tablet_replica.cc:442] T fd733d18b51e447a8dc0b20cd0c0408b P b3c245430dbb4631a28158377c91d819: This TxnStatusTablet replica's current role is: LEADER
I20260926 12:33:58.920112 18681 tablet_replica.cc:439] T fd733d18b51e447a8dc0b20cd0c0408b P 13f4dc3adf864c30a639b4240fd58dd3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3c245430dbb4631a28158377c91d819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } } }
I20260926 12:33:58.920217 18681 tablet_replica.cc:442] T fd733d18b51e447a8dc0b20cd0c0408b P 13f4dc3adf864c30a639b4240fd58dd3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:33:58.920887 18676 mvcc.cc:205] Tried to move back new op lower bound from 7333585055389143040 to 7333585055293751296. Current Snapshot: MvccSnapshot[applied={T|T < 7333585055389143040}]
I20260926 12:33:58.921051 18669 tablet_replica.cc:439] T fd733d18b51e447a8dc0b20cd0c0408b P d92e828418f149e4a847047d6fedc719: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b3c245430dbb4631a28158377c91d819" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } } }
I20260926 12:33:58.921177 18669 tablet_replica.cc:442] T fd733d18b51e447a8dc0b20cd0c0408b P d92e828418f149e4a847047d6fedc719: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 12:33:58.929245 18709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.925834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.10.98.1:35743)
0926 12:33:58.925932 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:58.925935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:58.925948 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:33:58.926098 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:58.926101 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:58.926143 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:58.926201 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:58.926421 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.926995 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.928176 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.928388 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.928499 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.928788 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:58.928861 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:58.928929 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:58.929031 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 12:33:58.929075 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 12:33:58.940189 18709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.936690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:43663 (local address 127.10.98.1:35743)
0926 12:33:58.936801 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:58.936805 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:58.936964 (+ 159us) server_negotiation.cc:407] Connection header received
0926 12:33:58.937078 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:58.937081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:58.937165 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:58.937238 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:58.937480 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.938238 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.939097 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.939417 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.939516 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.939758 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:58.939839 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:58.939905 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:58.939967 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 12:33:58.940014 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":39}
I20260926 12:33:58.940194 18716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:58.936612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:35743 (local address 127.10.98.1:43663)
0926 12:33:58.936884 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:58.936899 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:58.936994 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:58.937257 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:58.937259 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:58.937266 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:33:58.937388 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.937393 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.938342 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.938345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.938991 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:58.939000 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:58.939531 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:58.939533 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:58.939605 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:58.939648 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:33:58.939910 (+ 262us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:33:58.939914 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:33:58.939956 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 12:33:58.940016 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":14,"thread_start_us":101,"threads_started":1}
I20260926 12:33:58.946748 18448 raft_consensus.cc:1261] T 30806e83823f48c3a0f844209732ce29 P 13f4dc3adf864c30a639b4240fd58dd3 [term 1 FOLLOWER]: Refusing update from remote peer b3c245430dbb4631a28158377c91d819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:58.946748 18578 raft_consensus.cc:1261] T 30806e83823f48c3a0f844209732ce29 P d92e828418f149e4a847047d6fedc719 [term 1 FOLLOWER]: Refusing update from remote peer b3c245430dbb4631a28158377c91d819: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:58.947173 18712 consensus_queue.cc:1035] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:58.947346 18670 consensus_queue.cc:1035] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:58.952337 18675 mvcc.cc:205] Tried to move back new op lower bound from 7333585055522435072 to 7333585054662574080. Current Snapshot: MvccSnapshot[applied={T|T < 7333585055522435072}]
I20260926 12:33:58.986229 18151 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:58.986403 18151 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:58.988545 18151 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299: Sending DeleteTablet for 3 replicas of tablet 30806e83823f48c3a0f844209732ce29
I20260926 12:33:58.989135 18428 tablet_service.cc:1514] Processing DeleteTablet for tablet 30806e83823f48c3a0f844209732ce29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:58 UTC) from {username='oryx', principal='oryx/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:56120
I20260926 12:33:58.989223 18568 tablet_service.cc:1514] Processing DeleteTablet for tablet 30806e83823f48c3a0f844209732ce29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:58 UTC) from {username='oryx', principal='oryx/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:54422
I20260926 12:33:58.989463 18288 tablet_service.cc:1514] Processing DeleteTablet for tablet 30806e83823f48c3a0f844209732ce29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:58 UTC) from {username='oryx', principal='oryx/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:50814
I20260926 12:33:58.989686 18721 tablet_replica.cc:330] T 30806e83823f48c3a0f844209732ce29 P d92e828418f149e4a847047d6fedc719: stopping tablet replica
I20260926 12:33:58.989818 18721 raft_consensus.cc:2229] T 30806e83823f48c3a0f844209732ce29 P d92e828418f149e4a847047d6fedc719 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:58.989940 18721 raft_consensus.cc:2258] T 30806e83823f48c3a0f844209732ce29 P d92e828418f149e4a847047d6fedc719 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:58.990617 18721 ts_tablet_manager.cc:1905] T 30806e83823f48c3a0f844209732ce29 P d92e828418f149e4a847047d6fedc719: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:58.994642 18720 tablet_replica.cc:330] T 30806e83823f48c3a0f844209732ce29 P 13f4dc3adf864c30a639b4240fd58dd3: stopping tablet replica
I20260926 12:33:58.994782 18720 raft_consensus.cc:2229] T 30806e83823f48c3a0f844209732ce29 P 13f4dc3adf864c30a639b4240fd58dd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:58.995530 18721 ts_tablet_manager.cc:1918] T 30806e83823f48c3a0f844209732ce29 P d92e828418f149e4a847047d6fedc719: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:33:58.995610 18721 log.cc:1196] T 30806e83823f48c3a0f844209732ce29 P d92e828418f149e4a847047d6fedc719: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/30806e83823f48c3a0f844209732ce29
I20260926 12:33:58.995844 18721 ts_tablet_manager.cc:1939] T 30806e83823f48c3a0f844209732ce29 P d92e828418f149e4a847047d6fedc719: Deleting consensus metadata
I20260926 12:33:58.996578 18138 catalog_manager.cc:4841] TS d92e828418f149e4a847047d6fedc719 (127.10.98.3:38065): tablet 30806e83823f48c3a0f844209732ce29 (table test-table [id=86c87f6f753e4136aabc1fcdd5700a77]) successfully deleted
I20260926 12:33:58.997174 18720 raft_consensus.cc:2258] T 30806e83823f48c3a0f844209732ce29 P 13f4dc3adf864c30a639b4240fd58dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:58.997926 18720 ts_tablet_manager.cc:1905] T 30806e83823f48c3a0f844209732ce29 P 13f4dc3adf864c30a639b4240fd58dd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:59.000926 18720 ts_tablet_manager.cc:1918] T 30806e83823f48c3a0f844209732ce29 P 13f4dc3adf864c30a639b4240fd58dd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:33:59.001017 18720 log.cc:1196] T 30806e83823f48c3a0f844209732ce29 P 13f4dc3adf864c30a639b4240fd58dd3: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/30806e83823f48c3a0f844209732ce29
I20260926 12:33:59.001279 18722 tablet_replica.cc:330] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819: stopping tablet replica
I20260926 12:33:59.001303 18720 ts_tablet_manager.cc:1939] T 30806e83823f48c3a0f844209732ce29 P 13f4dc3adf864c30a639b4240fd58dd3: Deleting consensus metadata
I20260926 12:33:59.001453 18722 raft_consensus.cc:2229] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:33:59.001786 18722 raft_consensus.cc:2258] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:59.001999 18137 catalog_manager.cc:4841] TS 13f4dc3adf864c30a639b4240fd58dd3 (127.10.98.2:40745): tablet 30806e83823f48c3a0f844209732ce29 (table test-table [id=86c87f6f753e4136aabc1fcdd5700a77]) successfully deleted
I20260926 12:33:59.002530 18722 ts_tablet_manager.cc:1905] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:59.004958 18722 ts_tablet_manager.cc:1918] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:33:59.005028 18722 log.cc:1196] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-0/wal/wals/30806e83823f48c3a0f844209732ce29
I20260926 12:33:59.005280 18722 ts_tablet_manager.cc:1939] T 30806e83823f48c3a0f844209732ce29 P b3c245430dbb4631a28158377c91d819: Deleting consensus metadata
I20260926 12:33:59.005999 18137 catalog_manager.cc:4841] TS b3c245430dbb4631a28158377c91d819 (127.10.98.1:35743): tablet 30806e83823f48c3a0f844209732ce29 (table test-table [id=86c87f6f753e4136aabc1fcdd5700a77]) successfully deleted
Sep 26 12:33:59 dist-test-slave-t7tk krb5kdc[18106](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 12:33:59 dist-test-slave-t7tk krb5kdc[18106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:33:59 dist-test-slave-t7tk krb5kdc[18106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 12:33:59.024307 18701 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:33:59.017245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.10.98.62:37705)
0926 12:33:59.017379 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:59.017385 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:59.017617 (+ 232us) server_negotiation.cc:407] Connection header received
0926 12:33:59.017754 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:59.017756 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:59.017803 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:59.017875 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:59.018783 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.019331 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.020528 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.020791 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.020872 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:59.024164 (+ 3292us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41488: BlockingRecv error: recv got EOF from 127.0.0.1:41488 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 12:33:59 dist-test-slave-t7tk krb5kdc[18106](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426039, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM
I20260926 12:33:59.035097 18701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:59.025766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.10.98.62:37705)
0926 12:33:59.025895 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:59.025899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:59.025974 (+ 75us) server_negotiation.cc:407] Connection header received
0926 12:33:59.026082 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:59.026086 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:59.026140 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:59.026226 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:33:59.026746 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.027578 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.028434 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.028760 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.028877 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:59.031586 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:33:59.031590 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:33:59.031594 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:33:59.031647 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:33:59.033273 (+ 1626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:59.033801 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:59.033804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:59.033805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:59.033896 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:33:59.034236 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:33:59.034239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:33:59.034241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:33:59.034434 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:33:59.034540 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:59.034797 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 12:33:59.034947 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:33:59.038019 18151 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:33:59.038514 18151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:33:59.043347 18288 tablet_service.cc:1467] Processing CreateTablet for tablet abdb5ac600094973a4c2fc0d52453aab (DEFAULT_TABLE table=test-table [id=2638efda2b2d4f83b89db5a0eba29717]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:59.043347 18428 tablet_service.cc:1467] Processing CreateTablet for tablet abdb5ac600094973a4c2fc0d52453aab (DEFAULT_TABLE table=test-table [id=2638efda2b2d4f83b89db5a0eba29717]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:59.043514 18568 tablet_service.cc:1467] Processing CreateTablet for tablet abdb5ac600094973a4c2fc0d52453aab (DEFAULT_TABLE table=test-table [id=2638efda2b2d4f83b89db5a0eba29717]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:59.043641 18428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdb5ac600094973a4c2fc0d52453aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:59.043642 18288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdb5ac600094973a4c2fc0d52453aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:59.043740 18568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abdb5ac600094973a4c2fc0d52453aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:59.046114 18663 tablet_bootstrap.cc:492] T abdb5ac600094973a4c2fc0d52453aab P b3c245430dbb4631a28158377c91d819: Bootstrap starting.
I20260926 12:33:59.046280 18662 tablet_bootstrap.cc:492] T abdb5ac600094973a4c2fc0d52453aab P 13f4dc3adf864c30a639b4240fd58dd3: Bootstrap starting.
I20260926 12:33:59.047077 18662 tablet_bootstrap.cc:654] T abdb5ac600094973a4c2fc0d52453aab P 13f4dc3adf864c30a639b4240fd58dd3: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:59.047395 18663 tablet_bootstrap.cc:654] T abdb5ac600094973a4c2fc0d52453aab P b3c245430dbb4631a28158377c91d819: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:59.047628 18664 tablet_bootstrap.cc:492] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719: Bootstrap starting.
I20260926 12:33:59.048483 18664 tablet_bootstrap.cc:654] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:59.048683 18663 tablet_bootstrap.cc:492] T abdb5ac600094973a4c2fc0d52453aab P b3c245430dbb4631a28158377c91d819: No bootstrap required, opened a new log
I20260926 12:33:59.048764 18663 ts_tablet_manager.cc:1397] T abdb5ac600094973a4c2fc0d52453aab P b3c245430dbb4631a28158377c91d819: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:33:59.049219 18663 raft_consensus.cc:348] T abdb5ac600094973a4c2fc0d52453aab P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:59.049360 18663 raft_consensus.cc:374] T abdb5ac600094973a4c2fc0d52453aab P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:59.049398 18663 raft_consensus.cc:729] T abdb5ac600094973a4c2fc0d52453aab P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3c245430dbb4631a28158377c91d819, State: Initialized, Role: FOLLOWER
I20260926 12:33:59.049528 18663 consensus_queue.cc:260] T abdb5ac600094973a4c2fc0d52453aab P b3c245430dbb4631a28158377c91d819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:59.049808 18663 ts_tablet_manager.cc:1428] T abdb5ac600094973a4c2fc0d52453aab P b3c245430dbb4631a28158377c91d819: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:59.049997 18662 tablet_bootstrap.cc:492] T abdb5ac600094973a4c2fc0d52453aab P 13f4dc3adf864c30a639b4240fd58dd3: No bootstrap required, opened a new log
I20260926 12:33:59.050086 18662 ts_tablet_manager.cc:1397] T abdb5ac600094973a4c2fc0d52453aab P 13f4dc3adf864c30a639b4240fd58dd3: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 12:33:59.050447 18662 raft_consensus.cc:348] T abdb5ac600094973a4c2fc0d52453aab P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:59.050549 18662 raft_consensus.cc:374] T abdb5ac600094973a4c2fc0d52453aab P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:59.050577 18662 raft_consensus.cc:729] T abdb5ac600094973a4c2fc0d52453aab P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13f4dc3adf864c30a639b4240fd58dd3, State: Initialized, Role: FOLLOWER
I20260926 12:33:59.050709 18664 tablet_bootstrap.cc:492] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719: No bootstrap required, opened a new log
I20260926 12:33:59.050707 18662 consensus_queue.cc:260] T abdb5ac600094973a4c2fc0d52453aab P 13f4dc3adf864c30a639b4240fd58dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:59.050791 18664 ts_tablet_manager.cc:1397] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260926 12:33:59.050952 18662 ts_tablet_manager.cc:1428] T abdb5ac600094973a4c2fc0d52453aab P 13f4dc3adf864c30a639b4240fd58dd3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:59.051236 18664 raft_consensus.cc:348] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:59.051347 18664 raft_consensus.cc:374] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:59.051379 18664 raft_consensus.cc:729] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92e828418f149e4a847047d6fedc719, State: Initialized, Role: FOLLOWER
I20260926 12:33:59.051488 18664 consensus_queue.cc:260] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:59.051744 18664 ts_tablet_manager.cc:1428] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:59.460238 18669 raft_consensus.cc:482] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:33:59.460386 18669 raft_consensus.cc:504] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:59.461632 18669 leader_election.cc:288] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3c245430dbb4631a28158377c91d819 (127.10.98.1:35743), 13f4dc3adf864c30a639b4240fd58dd3 (127.10.98.2:40745)
I20260926 12:33:59.465674 18739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:59.462060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:53435 (local address 127.10.98.1:35743)
0926 12:33:59.462359 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:59.462364 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:59.462382 (+ 18us) server_negotiation.cc:407] Connection header received
0926 12:33:59.462511 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:59.462514 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:59.462566 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:59.462668 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:59.462934 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.463478 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.464594 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.465041 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.465190 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:59.465265 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:59.465473 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 12:33:59.465551 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":132,"threads_started":1}
I20260926 12:33:59.465720 18738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:59.461955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:35743 (local address 127.10.98.3:53435)
0926 12:33:59.462252 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:59.462268 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:59.462377 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:59.462681 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:59.462684 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:59.462704 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:59.462810 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:59.462815 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.463591 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.463594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:59.464472 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:59.464482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.465199 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.465203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:59.465302 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:59.465333 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 12:33:59.465424 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 12:33:59.465503 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"spinlock_wait_cycles":896,"thread_start_us":100,"threads_started":1}
I20260926 12:33:59.466176 18740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:59.462390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:40745 (local address 127.10.98.3:56825)
0926 12:33:59.462629 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:59.462643 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:59.462731 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:59.462961 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:59.462964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:59.462980 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:33:59.463103 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:59.463109 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.463881 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.463884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:59.464771 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:59.464782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.465752 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.465756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:59.465863 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:59.465886 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:33:59.465967 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 12:33:59.466039 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":59,"threads_started":1}
I20260926 12:33:59.466292 18741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:59.462496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:56825 (local address 127.10.98.2:40745)
0926 12:33:59.462733 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:59.462738 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:59.462756 (+ 18us) server_negotiation.cc:407] Connection header received
0926 12:33:59.462813 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:59.462816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:59.462866 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:59.462945 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:33:59.463220 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.463768 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.464905 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.465563 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.465690 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:59.465769 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:59.466057 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 12:33:59.466137 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":62,"threads_started":1}
I20260926 12:33:59.466236 18308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb5ac600094973a4c2fc0d52453aab" candidate_uuid: "d92e828418f149e4a847047d6fedc719" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3c245430dbb4631a28158377c91d819" is_pre_election: true
I20260926 12:33:59.466410 18308 raft_consensus.cc:2454] T abdb5ac600094973a4c2fc0d52453aab P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92e828418f149e4a847047d6fedc719 in term 0.
I20260926 12:33:59.466809 18448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb5ac600094973a4c2fc0d52453aab" candidate_uuid: "d92e828418f149e4a847047d6fedc719" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f4dc3adf864c30a639b4240fd58dd3" is_pre_election: true
I20260926 12:33:59.466815 18521 leader_election.cc:302] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3c245430dbb4631a28158377c91d819, d92e828418f149e4a847047d6fedc719; no voters:
I20260926 12:33:59.466949 18448 raft_consensus.cc:2454] T abdb5ac600094973a4c2fc0d52453aab P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92e828418f149e4a847047d6fedc719 in term 0.
I20260926 12:33:59.467067 18669 raft_consensus.cc:2781] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:33:59.467124 18669 raft_consensus.cc:482] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:33:59.467162 18669 raft_consensus.cc:3037] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:59.468163 18669 raft_consensus.cc:504] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:59.468489 18669 leader_election.cc:288] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 [CANDIDATE]: Term 1 election: Requested vote from peers b3c245430dbb4631a28158377c91d819 (127.10.98.1:35743), 13f4dc3adf864c30a639b4240fd58dd3 (127.10.98.2:40745)
I20260926 12:33:59.468922 18308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb5ac600094973a4c2fc0d52453aab" candidate_uuid: "d92e828418f149e4a847047d6fedc719" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3c245430dbb4631a28158377c91d819"
I20260926 12:33:59.468954 18448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abdb5ac600094973a4c2fc0d52453aab" candidate_uuid: "d92e828418f149e4a847047d6fedc719" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f4dc3adf864c30a639b4240fd58dd3"
I20260926 12:33:59.469058 18308 raft_consensus.cc:3037] T abdb5ac600094973a4c2fc0d52453aab P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:59.469074 18448 raft_consensus.cc:3037] T abdb5ac600094973a4c2fc0d52453aab P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:59.470065 18308 raft_consensus.cc:2454] T abdb5ac600094973a4c2fc0d52453aab P b3c245430dbb4631a28158377c91d819 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d92e828418f149e4a847047d6fedc719 in term 1.
I20260926 12:33:59.470077 18448 raft_consensus.cc:2454] T abdb5ac600094973a4c2fc0d52453aab P 13f4dc3adf864c30a639b4240fd58dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d92e828418f149e4a847047d6fedc719 in term 1.
I20260926 12:33:59.470419 18521 leader_election.cc:302] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13f4dc3adf864c30a639b4240fd58dd3, d92e828418f149e4a847047d6fedc719; no voters:
I20260926 12:33:59.470594 18669 raft_consensus.cc:2781] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:59.470795 18669 raft_consensus.cc:686] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 [term 1 LEADER]: Becoming Leader. State: Replica: d92e828418f149e4a847047d6fedc719, State: Running, Role: LEADER
I20260926 12:33:59.471071 18669 consensus_queue.cc:237] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:59.473524 18151 catalog_manager.cc:5495] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 reported cstate change: term changed from 0 to 1, leader changed from <none> to d92e828418f149e4a847047d6fedc719 (127.10.98.3). New cstate: current_term: 1 leader_uuid: "d92e828418f149e4a847047d6fedc719" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } health_report { overall_health: HEALTHY } } }
I20260926 12:33:59.508252 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:59.504736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46544 (local address 127.10.98.3:38065)
0926 12:33:59.504950 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:59.504954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:59.504967 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:33:59.505046 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:59.505048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:59.505130 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:59.505236 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:59.505433 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.505953 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.506825 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.507045 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.507132 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:59.507409 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:59.507865 (+ 456us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:59.507970 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:59.508041 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 12:33:59.508100 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":66,"threads_started":1}
I20260926 12:33:59.510980 18308 raft_consensus.cc:1261] T abdb5ac600094973a4c2fc0d52453aab P b3c245430dbb4631a28158377c91d819 [term 1 FOLLOWER]: Refusing update from remote peer d92e828418f149e4a847047d6fedc719: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:59.510984 18448 raft_consensus.cc:1261] T abdb5ac600094973a4c2fc0d52453aab P 13f4dc3adf864c30a639b4240fd58dd3 [term 1 FOLLOWER]: Refusing update from remote peer d92e828418f149e4a847047d6fedc719: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:59.511512 18669 consensus_queue.cc:1035] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:59.511749 18742 consensus_queue.cc:1035] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:59.522927 18151 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:59.523101 18151 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:59.525506 18151 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299: Sending DeleteTablet for 3 replicas of tablet abdb5ac600094973a4c2fc0d52453aab
I20260926 12:33:59.526124 18288 tablet_service.cc:1514] Processing DeleteTablet for tablet abdb5ac600094973a4c2fc0d52453aab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:59 UTC) from {username='oryx', principal='oryx/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:50814
I20260926 12:33:59.526178 18428 tablet_service.cc:1514] Processing DeleteTablet for tablet abdb5ac600094973a4c2fc0d52453aab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:59 UTC) from {username='oryx', principal='oryx/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:56120
I20260926 12:33:59.526458 18568 tablet_service.cc:1514] Processing DeleteTablet for tablet abdb5ac600094973a4c2fc0d52453aab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:59 UTC) from {username='oryx', principal='oryx/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:54422
I20260926 12:33:59.526613 18757 tablet_replica.cc:330] T abdb5ac600094973a4c2fc0d52453aab P 13f4dc3adf864c30a639b4240fd58dd3: stopping tablet replica
I20260926 12:33:59.526749 18757 raft_consensus.cc:2229] T abdb5ac600094973a4c2fc0d52453aab P 13f4dc3adf864c30a639b4240fd58dd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:59.526889 18757 raft_consensus.cc:2258] T abdb5ac600094973a4c2fc0d52453aab P 13f4dc3adf864c30a639b4240fd58dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:59.527344 18758 tablet_replica.cc:330] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719: stopping tablet replica
I20260926 12:33:59.527473 18758 raft_consensus.cc:2229] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:33:59.527463 18151 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41496:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 12:33:59.527595 18757 ts_tablet_manager.cc:1905] T abdb5ac600094973a4c2fc0d52453aab P 13f4dc3adf864c30a639b4240fd58dd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:59.527783 18758 raft_consensus.cc:2258] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 12:33:59.527899 18151 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:33:59.528463 18758 ts_tablet_manager.cc:1905] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:59.530328 18757 ts_tablet_manager.cc:1918] T abdb5ac600094973a4c2fc0d52453aab P 13f4dc3adf864c30a639b4240fd58dd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:33:59.530405 18757 log.cc:1196] T abdb5ac600094973a4c2fc0d52453aab P 13f4dc3adf864c30a639b4240fd58dd3: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/abdb5ac600094973a4c2fc0d52453aab
I20260926 12:33:59.530692 18757 ts_tablet_manager.cc:1939] T abdb5ac600094973a4c2fc0d52453aab P 13f4dc3adf864c30a639b4240fd58dd3: Deleting consensus metadata
I20260926 12:33:59.531435 18137 catalog_manager.cc:4841] TS 13f4dc3adf864c30a639b4240fd58dd3 (127.10.98.2:40745): tablet abdb5ac600094973a4c2fc0d52453aab (table test-table [id=2638efda2b2d4f83b89db5a0eba29717]) successfully deleted
I20260926 12:33:59.531761 18756 tablet_replica.cc:330] T abdb5ac600094973a4c2fc0d52453aab P b3c245430dbb4631a28158377c91d819: stopping tablet replica
I20260926 12:33:59.531893 18756 raft_consensus.cc:2229] T abdb5ac600094973a4c2fc0d52453aab P b3c245430dbb4631a28158377c91d819 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:59.532094 18756 raft_consensus.cc:2258] T abdb5ac600094973a4c2fc0d52453aab P b3c245430dbb4631a28158377c91d819 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:59.532848 18756 ts_tablet_manager.cc:1905] T abdb5ac600094973a4c2fc0d52453aab P b3c245430dbb4631a28158377c91d819: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:59.533819 18428 tablet_service.cc:1467] Processing CreateTablet for tablet 7888b4953a604ec4b23c7bbf66356801 (DEFAULT_TABLE table=test-table [id=eb3b4ba4edd140a5ab30815af230ddd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:59.533859 18288 tablet_service.cc:1467] Processing CreateTablet for tablet 7888b4953a604ec4b23c7bbf66356801 (DEFAULT_TABLE table=test-table [id=eb3b4ba4edd140a5ab30815af230ddd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:59.533936 18568 tablet_service.cc:1467] Processing CreateTablet for tablet 7888b4953a604ec4b23c7bbf66356801 (DEFAULT_TABLE table=test-table [id=eb3b4ba4edd140a5ab30815af230ddd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:33:59.534058 18428 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7888b4953a604ec4b23c7bbf66356801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:59.534058 18288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7888b4953a604ec4b23c7bbf66356801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:59.534173 18568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7888b4953a604ec4b23c7bbf66356801. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:33:59.536316 18662 tablet_bootstrap.cc:492] T 7888b4953a604ec4b23c7bbf66356801 P 13f4dc3adf864c30a639b4240fd58dd3: Bootstrap starting.
I20260926 12:33:59.536448 18663 tablet_bootstrap.cc:492] T 7888b4953a604ec4b23c7bbf66356801 P b3c245430dbb4631a28158377c91d819: Bootstrap starting.
I20260926 12:33:59.536970 18758 ts_tablet_manager.cc:1918] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:33:59.537040 18758 log.cc:1196] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/abdb5ac600094973a4c2fc0d52453aab
I20260926 12:33:59.537165 18662 tablet_bootstrap.cc:654] T 7888b4953a604ec4b23c7bbf66356801 P 13f4dc3adf864c30a639b4240fd58dd3: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:59.537307 18758 ts_tablet_manager.cc:1939] T abdb5ac600094973a4c2fc0d52453aab P d92e828418f149e4a847047d6fedc719: Deleting consensus metadata
I20260926 12:33:59.537736 18664 tablet_bootstrap.cc:492] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719: Bootstrap starting.
I20260926 12:33:59.537782 18663 tablet_bootstrap.cc:654] T 7888b4953a604ec4b23c7bbf66356801 P b3c245430dbb4631a28158377c91d819: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:59.538280 18138 catalog_manager.cc:4841] TS d92e828418f149e4a847047d6fedc719 (127.10.98.3:38065): tablet abdb5ac600094973a4c2fc0d52453aab (table test-table [id=2638efda2b2d4f83b89db5a0eba29717]) successfully deleted
I20260926 12:33:59.538609 18664 tablet_bootstrap.cc:654] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719: Neither blocks nor log segments found. Creating new log.
I20260926 12:33:59.539332 18663 tablet_bootstrap.cc:492] T 7888b4953a604ec4b23c7bbf66356801 P b3c245430dbb4631a28158377c91d819: No bootstrap required, opened a new log
I20260926 12:33:59.539413 18663 ts_tablet_manager.cc:1397] T 7888b4953a604ec4b23c7bbf66356801 P b3c245430dbb4631a28158377c91d819: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 12:33:59.539676 18664 tablet_bootstrap.cc:492] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719: No bootstrap required, opened a new log
I20260926 12:33:59.539757 18664 ts_tablet_manager.cc:1397] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 12:33:59.539786 18663 raft_consensus.cc:348] T 7888b4953a604ec4b23c7bbf66356801 P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:59.539892 18663 raft_consensus.cc:374] T 7888b4953a604ec4b23c7bbf66356801 P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:59.539920 18663 raft_consensus.cc:729] T 7888b4953a604ec4b23c7bbf66356801 P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3c245430dbb4631a28158377c91d819, State: Initialized, Role: FOLLOWER
I20260926 12:33:59.540035 18663 consensus_queue.cc:260] T 7888b4953a604ec4b23c7bbf66356801 P b3c245430dbb4631a28158377c91d819 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:59.540140 18664 raft_consensus.cc:348] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:59.540235 18664 raft_consensus.cc:374] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:59.540262 18664 raft_consensus.cc:729] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d92e828418f149e4a847047d6fedc719, State: Initialized, Role: FOLLOWER
I20260926 12:33:59.540306 18663 ts_tablet_manager.cc:1428] T 7888b4953a604ec4b23c7bbf66356801 P b3c245430dbb4631a28158377c91d819: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 12:33:59.540382 18664 consensus_queue.cc:260] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:59.540616 18664 ts_tablet_manager.cc:1428] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:33:59.541482 18756 ts_tablet_manager.cc:1918] T abdb5ac600094973a4c2fc0d52453aab P b3c245430dbb4631a28158377c91d819: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:33:59.541548 18756 log.cc:1196] T abdb5ac600094973a4c2fc0d52453aab P b3c245430dbb4631a28158377c91d819: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-0/wal/wals/abdb5ac600094973a4c2fc0d52453aab
I20260926 12:33:59.541896 18756 ts_tablet_manager.cc:1939] T abdb5ac600094973a4c2fc0d52453aab P b3c245430dbb4631a28158377c91d819: Deleting consensus metadata
I20260926 12:33:59.542687 18137 catalog_manager.cc:4841] TS b3c245430dbb4631a28158377c91d819 (127.10.98.1:35743): tablet abdb5ac600094973a4c2fc0d52453aab (table test-table [id=2638efda2b2d4f83b89db5a0eba29717]) successfully deleted
I20260926 12:33:59.543099 18662 tablet_bootstrap.cc:492] T 7888b4953a604ec4b23c7bbf66356801 P 13f4dc3adf864c30a639b4240fd58dd3: No bootstrap required, opened a new log
I20260926 12:33:59.543185 18662 ts_tablet_manager.cc:1397] T 7888b4953a604ec4b23c7bbf66356801 P 13f4dc3adf864c30a639b4240fd58dd3: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260926 12:33:59.543601 18662 raft_consensus.cc:348] T 7888b4953a604ec4b23c7bbf66356801 P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:59.543706 18662 raft_consensus.cc:374] T 7888b4953a604ec4b23c7bbf66356801 P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:33:59.543735 18662 raft_consensus.cc:729] T 7888b4953a604ec4b23c7bbf66356801 P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13f4dc3adf864c30a639b4240fd58dd3, State: Initialized, Role: FOLLOWER
I20260926 12:33:59.543859 18662 consensus_queue.cc:260] T 7888b4953a604ec4b23c7bbf66356801 P 13f4dc3adf864c30a639b4240fd58dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:59.544096 18662 ts_tablet_manager.cc:1428] T 7888b4953a604ec4b23c7bbf66356801 P 13f4dc3adf864c30a639b4240fd58dd3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 12:33:59.578325 18753 raft_consensus.cc:482] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:33:59.578516 18753 raft_consensus.cc:504] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:59.578987 18753 leader_election.cc:288] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3c245430dbb4631a28158377c91d819 (127.10.98.1:35743), 13f4dc3adf864c30a639b4240fd58dd3 (127.10.98.2:40745)
I20260926 12:33:59.579385 18308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7888b4953a604ec4b23c7bbf66356801" candidate_uuid: "d92e828418f149e4a847047d6fedc719" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3c245430dbb4631a28158377c91d819" is_pre_election: true
I20260926 12:33:59.579417 18448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7888b4953a604ec4b23c7bbf66356801" candidate_uuid: "d92e828418f149e4a847047d6fedc719" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f4dc3adf864c30a639b4240fd58dd3" is_pre_election: true
I20260926 12:33:59.579530 18308 raft_consensus.cc:2454] T 7888b4953a604ec4b23c7bbf66356801 P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92e828418f149e4a847047d6fedc719 in term 0.
I20260926 12:33:59.579596 18448 raft_consensus.cc:2454] T 7888b4953a604ec4b23c7bbf66356801 P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d92e828418f149e4a847047d6fedc719 in term 0.
I20260926 12:33:59.579807 18521 leader_election.cc:302] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3c245430dbb4631a28158377c91d819, d92e828418f149e4a847047d6fedc719; no voters:
I20260926 12:33:59.579957 18753 raft_consensus.cc:2781] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:33:59.580010 18753 raft_consensus.cc:482] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:33:59.580052 18753 raft_consensus.cc:3037] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:59.581066 18753 raft_consensus.cc:504] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:59.581420 18753 leader_election.cc:288] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 [CANDIDATE]: Term 1 election: Requested vote from peers b3c245430dbb4631a28158377c91d819 (127.10.98.1:35743), 13f4dc3adf864c30a639b4240fd58dd3 (127.10.98.2:40745)
I20260926 12:33:59.581766 18308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7888b4953a604ec4b23c7bbf66356801" candidate_uuid: "d92e828418f149e4a847047d6fedc719" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3c245430dbb4631a28158377c91d819"
I20260926 12:33:59.581826 18448 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7888b4953a604ec4b23c7bbf66356801" candidate_uuid: "d92e828418f149e4a847047d6fedc719" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13f4dc3adf864c30a639b4240fd58dd3"
I20260926 12:33:59.581887 18308 raft_consensus.cc:3037] T 7888b4953a604ec4b23c7bbf66356801 P b3c245430dbb4631a28158377c91d819 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:59.581949 18448 raft_consensus.cc:3037] T 7888b4953a604ec4b23c7bbf66356801 P 13f4dc3adf864c30a639b4240fd58dd3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:33:59.583006 18448 raft_consensus.cc:2454] T 7888b4953a604ec4b23c7bbf66356801 P 13f4dc3adf864c30a639b4240fd58dd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d92e828418f149e4a847047d6fedc719 in term 1.
I20260926 12:33:59.583006 18308 raft_consensus.cc:2454] T 7888b4953a604ec4b23c7bbf66356801 P b3c245430dbb4631a28158377c91d819 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d92e828418f149e4a847047d6fedc719 in term 1.
I20260926 12:33:59.583276 18521 leader_election.cc:302] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3c245430dbb4631a28158377c91d819, d92e828418f149e4a847047d6fedc719; no voters:
I20260926 12:33:59.583441 18753 raft_consensus.cc:2781] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:33:59.583540 18753 raft_consensus.cc:686] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 [term 1 LEADER]: Becoming Leader. State: Replica: d92e828418f149e4a847047d6fedc719, State: Running, Role: LEADER
I20260926 12:33:59.583799 18753 consensus_queue.cc:237] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } }
I20260926 12:33:59.585449 18150 catalog_manager.cc:5495] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 reported cstate change: term changed from 0 to 1, leader changed from <none> to d92e828418f149e4a847047d6fedc719 (127.10.98.3). New cstate: current_term: 1 leader_uuid: "d92e828418f149e4a847047d6fedc719" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d92e828418f149e4a847047d6fedc719" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38065 } health_report { overall_health: HEALTHY } } }
I20260926 12:33:59.608932 18739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:59.605393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:41763 (local address 127.10.98.1:35743)
0926 12:33:59.605525 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:59.605530 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:59.605667 (+ 137us) server_negotiation.cc:407] Connection header received
0926 12:33:59.605816 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:59.605818 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:59.605868 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:59.605931 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:59.606179 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.606720 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.607921 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.608149 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.608230 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:59.608535 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:59.608605 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:59.608651 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:59.608765 (+ 114us) server_negotiation.cc:299] Negotiation successful
0926 12:33:59.608817 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:33:59.608973 18767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:59.605273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:35743 (local address 127.10.98.3:41763)
0926 12:33:59.605585 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:59.605601 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:59.605709 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:59.605956 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:59.605959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:59.605967 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:33:59.606090 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:59.606096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.606820 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.606823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:59.607791 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:59.607802 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.608252 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.608256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:59.608355 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:59.608410 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:33:59.608674 (+ 264us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:33:59.608679 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 12:33:59.608739 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 12:33:59.608819 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":108,"threads_started":1}
I20260926 12:33:59.615175 18308 raft_consensus.cc:1261] T 7888b4953a604ec4b23c7bbf66356801 P b3c245430dbb4631a28158377c91d819 [term 1 FOLLOWER]: Refusing update from remote peer d92e828418f149e4a847047d6fedc719: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:59.615200 18448 raft_consensus.cc:1261] T 7888b4953a604ec4b23c7bbf66356801 P 13f4dc3adf864c30a639b4240fd58dd3 [term 1 FOLLOWER]: Refusing update from remote peer d92e828418f149e4a847047d6fedc719: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:33:59.615541 18753 consensus_queue.cc:1035] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13f4dc3adf864c30a639b4240fd58dd3" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:59.615720 18669 consensus_queue.cc:1035] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3c245430dbb4631a28158377c91d819" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:33:59.634716 18745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:59.630517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:37183 (local address 127.10.98.3:38065)
0926 12:33:59.630651 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 12:33:59.630656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:33:59.630820 (+ 164us) server_negotiation.cc:407] Connection header received
0926 12:33:59.630921 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:33:59.630924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:33:59.630974 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:33:59.631048 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:33:59.631357 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.632218 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.633480 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.633823 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.633968 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:59.634196 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:33:59.634314 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:33:59.634375 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 12:33:59.634517 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 12:33:59.634570 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:33:59.634743 18771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:59.630465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:38065 (local address 127.10.98.1:37183)
0926 12:33:59.630740 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:59.630754 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:33:59.630841 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:33:59.631074 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:33:59.631077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:33:59.631083 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 12:33:59.631211 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:59.631217 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.632492 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.632496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:59.633353 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:33:59.633364 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:33:59.633969 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:33:59.633972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:33:59.634054 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:33:59.634083 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 12:33:59.634416 (+ 333us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 12:33:59.634425 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 12:33:59.634488 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 12:33:59.634589 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":113,"threads_started":1}
I20260926 12:33:59.657410 18150 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:59.657598 18150 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41496:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:33:59.659934 18150 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1de7308f77074ee3872f1dd01d351299: Sending DeleteTablet for 3 replicas of tablet 7888b4953a604ec4b23c7bbf66356801
I20260926 12:33:59.660629 18568 tablet_service.cc:1514] Processing DeleteTablet for tablet 7888b4953a604ec4b23c7bbf66356801 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:59 UTC) from {username='oryx', principal='oryx/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:54422
I20260926 12:33:59.660636 18428 tablet_service.cc:1514] Processing DeleteTablet for tablet 7888b4953a604ec4b23c7bbf66356801 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:33:59 UTC) from {username='oryx', principal='oryx/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:56120
I20260926 12:33:59.660859 18758 tablet_replica.cc:330] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719: stopping tablet replica
I20260926 12:33:59.660929 18757 tablet_replica.cc:330] T 7888b4953a604ec4b23c7bbf66356801 P 13f4dc3adf864c30a639b4240fd58dd3: stopping tablet replica
I20260926 12:33:59.661000 18758 raft_consensus.cc:2229] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:33:59.661062 18757 raft_consensus.cc:2229] T 7888b4953a604ec4b23c7bbf66356801 P 13f4dc3adf864c30a639b4240fd58dd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:33:59.661207 18757 raft_consensus.cc:2258] T 7888b4953a604ec4b23c7bbf66356801 P 13f4dc3adf864c30a639b4240fd58dd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:59.661255 18758 raft_consensus.cc:2258] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:33:59.661926 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 18223
I20260926 12:33:59.662066 18757 ts_tablet_manager.cc:1905] T 7888b4953a604ec4b23c7bbf66356801 P 13f4dc3adf864c30a639b4240fd58dd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:59.662066 18758 ts_tablet_manager.cc:1905] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:33:59.664740 18758 ts_tablet_manager.cc:1918] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:33:59.664819 18758 log.cc:1196] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/7888b4953a604ec4b23c7bbf66356801
I20260926 12:33:59.665628 18757 ts_tablet_manager.cc:1918] T 7888b4953a604ec4b23c7bbf66356801 P 13f4dc3adf864c30a639b4240fd58dd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 12:33:59.665693 18758 ts_tablet_manager.cc:1939] T 7888b4953a604ec4b23c7bbf66356801 P d92e828418f149e4a847047d6fedc719: Deleting consensus metadata
I20260926 12:33:59.665712 18757 log.cc:1196] T 7888b4953a604ec4b23c7bbf66356801 P 13f4dc3adf864c30a639b4240fd58dd3: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/7888b4953a604ec4b23c7bbf66356801
I20260926 12:33:59.665938 18757 ts_tablet_manager.cc:1939] T 7888b4953a604ec4b23c7bbf66356801 P 13f4dc3adf864c30a639b4240fd58dd3: Deleting consensus metadata
I20260926 12:33:59.666539 18138 catalog_manager.cc:4841] TS d92e828418f149e4a847047d6fedc719 (127.10.98.3:38065): tablet 7888b4953a604ec4b23c7bbf66356801 (table test-table [id=eb3b4ba4edd140a5ab30815af230ddd8]) successfully deleted
I20260926 12:33:59.666594 18137 catalog_manager.cc:4841] TS 13f4dc3adf864c30a639b4240fd58dd3 (127.10.98.2:40745): tablet 7888b4953a604ec4b23c7bbf66356801 (table test-table [id=eb3b4ba4edd140a5ab30815af230ddd8]) successfully deleted
W20260926 12:33:59.671592 18137 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.98.1:35743 (error 108)
I20260926 12:33:59.671684 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 18363
I20260926 12:33:59.678082 18774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:33:59.677674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:35743 (local address 127.0.0.1:52956)
0926 12:33:59.677892 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0926 12:33:59.677969 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.1:35743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":64,"threads_started":1}
W20260926 12:33:59.678331 18137 catalog_manager.cc:4584] TS b3c245430dbb4631a28158377c91d819 (127.10.98.1:35743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7888b4953a604ec4b23c7bbf66356801: Network error: Client connection negotiation failed: client connection to 127.10.98.1:35743: connect: Connection refused (error 111)
I20260926 12:33:59.679643 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 18503
I20260926 12:33:59.685487 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 18119
2026-09-26T12:33:59Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3488 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:33:59 dist-test-slave-t7tk krb5kdc[18777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:33:59 dist-test-slave-t7tk krb5kdc[18777](info): set up 2 sockets
Sep 26 12:33:59 dist-test-slave-t7tk krb5kdc[18777](info): commencing operation
krb5kdc: starting...
W20260926 12:34:00.663326 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:34:00 dist-test-slave-t7tk krb5kdc[18777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426040, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:34:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:34:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:34:00.788914 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:38951
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:38951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:34:00.899051 18790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:34:00.899322 18790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:34:00.899363 18790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:34:00.903458 18790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:34:00.903528 18790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:34:00.903548 18790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:34:00.903563 18790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:34:00.903577 18790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:34:00.909061 18790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:38951
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:38951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:34:00.910185 18790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:34:00.911232 18790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:34:00.919813 18796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:00.919876 18790 server_base.cc:1034] running on GCE node
W20260926 12:34:00.919891 18798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:00.919792 18795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:00.920550 18790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:34:00.921746 18790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:34:00.922915 18790 hybrid_clock.cc:648] HybridClock initialized: now 1790426040922887 us; error 42 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T12:34:00Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1251 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:34:00 dist-test-slave-t7tk krb5kdc[18803](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:34:00 dist-test-slave-t7tk krb5kdc[18803](info): set up 2 sockets
Sep 26 12:34:00 dist-test-slave-t7tk krb5kdc[18803](info): commencing operation
krb5kdc: starting...
W20260926 12:34:01.891119 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:34:01 dist-test-slave-t7tk krb5kdc[18803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:34:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:34:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:34:02.016625 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:34931
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:36457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:34931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:34:02.127866 18816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:34:02.128165 18816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:34:02.128221 18816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:34:02.132315 18816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:34:02.132386 18816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:34:02.132411 18816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:34:02.132432 18816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:34:02.132452 18816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:34:02.137918 18816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:36457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:34931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:34931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:34:02.139038 18816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:34:02.140069 18816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:34:02.149042 18824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:02.149294 18816 server_base.cc:1034] running on GCE node
W20260926 12:34:02.149029 18821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:02.148995 18822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:02.150135 18816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:34:02.151326 18816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:34:02.152516 18816 hybrid_clock.cc:648] HybridClock initialized: now 1790426042152486 us; error 43 us; skew 500 ppm
Sep 26 12:34:02 dist-test-slave-t7tk krb5kdc[18803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426042, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:34:02.155488 18816 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:34:02.156445 18816 webserver.cc:460] Webserver started at http://127.10.98.62:37305/ using document root <none> and password file <none>
I20260926 12:34:02.157001 18816 fs_manager.cc:359] Metadata directory not provided
I20260926 12:34:02.157068 18816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:34:02.157292 18816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:34:02.159394 18816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "94feb09e1ad2492eaae6bca6a4de85b0"
format_stamp: "Formatted at 2026-09-26 12:34:02 on dist-test-slave-t7tk"
server_key: "0f677b1e10a35d397063636da691ef64"
server_key_iv: "ef3019e4fa7989553693adf9f19c0fe3"
server_key_version: "encryptionkey@0"
I20260926 12:34:02.159981 18816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "94feb09e1ad2492eaae6bca6a4de85b0"
format_stamp: "Formatted at 2026-09-26 12:34:02 on dist-test-slave-t7tk"
server_key: "0f677b1e10a35d397063636da691ef64"
server_key_iv: "ef3019e4fa7989553693adf9f19c0fe3"
server_key_version: "encryptionkey@0"
I20260926 12:34:02.163921 18816 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:34:02.166581 18831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:02.168120 18816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:34:02.168279 18816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "94feb09e1ad2492eaae6bca6a4de85b0"
format_stamp: "Formatted at 2026-09-26 12:34:02 on dist-test-slave-t7tk"
server_key: "0f677b1e10a35d397063636da691ef64"
server_key_iv: "ef3019e4fa7989553693adf9f19c0fe3"
server_key_version: "encryptionkey@0"
I20260926 12:34:02.168397 18816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:34:02.180169 18816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:34:02.183531 18816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:34:02.183739 18816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:34:02.191435 18816 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:34931
I20260926 12:34:02.191478 18883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:34931 every 8 connection(s)
I20260926 12:34:02.192401 18816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:34:02.193186 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 18816
I20260926 12:34:02.193298 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:34:02.193564 10632 external_mini_cluster.cc:1428] Setting key 254d51343a8977135a4949478cbbc54e
I20260926 12:34:02.195577 18884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:34:02.202672 18884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0: Bootstrap starting.
Sep 26 12:34:02 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:34:02.205591 18884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0: Neither blocks nor log segments found. Creating new log.
I20260926 12:34:02.206591 18884 log.cc:824] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0: Log is configured to *not* fsync() on all Append() calls
I20260926 12:34:02.209161 18884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0: No bootstrap required, opened a new log
I20260926 12:34:02.209826 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:02.194972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60592 (local address 127.10.98.62:34931)
0926 12:34:02.195386 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:02.195404 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:02.195448 (+ 44us) server_negotiation.cc:407] Connection header received
0926 12:34:02.196317 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:02.196343 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:02.196761 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:02.197177 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:02.197984 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:02.199483 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:02.200700 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:02.201462 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:02.201625 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:02.204079 (+ 2454us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:34:02.204086 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:34:02.204103 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:34:02.204146 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:34:02.206392 (+ 2246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:34:02.206911 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:34:02.206918 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:34:02.206924 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:34:02.207084 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:34:02.207409 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:34:02.207413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:34:02.207414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:34:02.207979 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:34:02.208171 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:02.208457 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 12:34:02.208793 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":114,"threads_started":1}
I20260926 12:34:02.212895 18884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94feb09e1ad2492eaae6bca6a4de85b0" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 34931 } }
I20260926 12:34:02.213155 18884 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:34:02.213214 18884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94feb09e1ad2492eaae6bca6a4de85b0, State: Initialized, Role: FOLLOWER
I20260926 12:34:02.213579 18884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94feb09e1ad2492eaae6bca6a4de85b0" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 34931 } }
I20260926 12:34:02.213690 18884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:34:02.213734 18884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:34:02.213802 18884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:34:02.214776 18884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94feb09e1ad2492eaae6bca6a4de85b0" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 34931 } }
I20260926 12:34:02.215116 18884 leader_election.cc:302] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94feb09e1ad2492eaae6bca6a4de85b0; no voters:
I20260926 12:34:02.215408 18884 leader_election.cc:288] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:34:02.215536 18889 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:34:02.215737 18889 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [term 1 LEADER]: Becoming Leader. State: Replica: 94feb09e1ad2492eaae6bca6a4de85b0, State: Running, Role: LEADER
I20260926 12:34:02.216024 18889 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94feb09e1ad2492eaae6bca6a4de85b0" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 34931 } }
I20260926 12:34:02.216521 18884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:34:02.217644 18891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94feb09e1ad2492eaae6bca6a4de85b0. Latest consensus state: current_term: 1 leader_uuid: "94feb09e1ad2492eaae6bca6a4de85b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94feb09e1ad2492eaae6bca6a4de85b0" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 34931 } } }
I20260926 12:34:02.217783 18891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [sys.catalog]: This master's current role is: LEADER
I20260926 12:34:02.217645 18890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94feb09e1ad2492eaae6bca6a4de85b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94feb09e1ad2492eaae6bca6a4de85b0" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 34931 } } }
I20260926 12:34:02.217888 18890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [sys.catalog]: This master's current role is: LEADER
I20260926 12:34:02.218210 18898 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:34:02.220870 18898 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:34:02.227065 18898 catalog_manager.cc:1347] Generated new cluster ID: 6014278bcfdb46e785bd7f279e260bad
I20260926 12:34:02.227162 18898 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:34:02.261394 18898 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:34:02.262401 18898 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:34:02.269513 18898 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0: Generated new TSK 0
I20260926 12:34:02.270242 18898 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
I20260926 12:34:02.304505 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:34931
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:36457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:34:02.416580 18910 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:34:02.416908 18910 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:34:02.416967 18910 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:34:02.421299 18910 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:34:02.421368 18910 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:34:02.421432 18910 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:34:02.427028 18910 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:36457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:34931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18910
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:34:02.428154 18910 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:34:02.429297 18910 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:34:02.439373 18915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:02.439375 18918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:02.439446 18916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:02.439628 18910 server_base.cc:1034] running on GCE node
I20260926 12:34:02.440065 18910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:34:02.440713 18910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:34:02.441916 18910 hybrid_clock.cc:648] HybridClock initialized: now 1790426042441896 us; error 32 us; skew 500 ppm
Sep 26 12:34:02 dist-test-slave-t7tk krb5kdc[18803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426042, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:34:02.444801 18910 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:34:02.445717 18910 webserver.cc:460] Webserver started at http://127.10.98.1:33915/ using document root <none> and password file <none>
I20260926 12:34:02.446316 18910 fs_manager.cc:359] Metadata directory not provided
I20260926 12:34:02.446404 18910 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:34:02.446624 18910 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:34:02.448818 18910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "8ede2d7d28af463699a35965b0553b7f"
format_stamp: "Formatted at 2026-09-26 12:34:02 on dist-test-slave-t7tk"
server_key: "6eb2eb97432c932040069aef34548755"
server_key_iv: "422e62a1e38c2420cd8e4e419cc30de6"
server_key_version: "encryptionkey@0"
I20260926 12:34:02.449388 18910 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "8ede2d7d28af463699a35965b0553b7f"
format_stamp: "Formatted at 2026-09-26 12:34:02 on dist-test-slave-t7tk"
server_key: "6eb2eb97432c932040069aef34548755"
server_key_iv: "422e62a1e38c2420cd8e4e419cc30de6"
server_key_version: "encryptionkey@0"
I20260926 12:34:02.453186 18910 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:34:02.456017 18925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:02.457331 18910 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:34:02.457442 18910 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "8ede2d7d28af463699a35965b0553b7f"
format_stamp: "Formatted at 2026-09-26 12:34:02 on dist-test-slave-t7tk"
server_key: "6eb2eb97432c932040069aef34548755"
server_key_iv: "422e62a1e38c2420cd8e4e419cc30de6"
server_key_version: "encryptionkey@0"
I20260926 12:34:02.457553 18910 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:34:02.472954 18910 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:34:02.476369 18910 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:34:02.476559 18910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:34:02.477218 18910 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:34:02.478106 18910 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:34:02.478154 18910 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:02.478209 18910 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:34:02.478235 18910 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:02.487504 18910 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:42487
I20260926 12:34:02.487543 19038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:42487 every 8 connection(s)
I20260926 12:34:02.488418 18910 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:34:02.491066 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 18910
I20260926 12:34:02.491206 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:34:02.491511 10632 external_mini_cluster.cc:1428] Setting key 4498c1bd6906b90a6a2cb0c51e7ead7f
Sep 26 12:34:02 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426042, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.98.2@KRBTEST.COM; defaulting to no policy
I20260926 12:34:02.502669 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:02.490269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:60449 (local address 127.10.98.62:34931)
0926 12:34:02.490467 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:02.490471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:02.491363 (+ 892us) server_negotiation.cc:407] Connection header received
0926 12:34:02.492789 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:02.492793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:02.492845 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:02.492918 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:02.494177 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:02.494705 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:02.495957 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:02.496295 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:02.496401 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:02.499091 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:34:02.499095 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:34:02.499098 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:34:02.499130 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:34:02.500617 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:34:02.501160 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:34:02.501162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:34:02.501164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:34:02.501241 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:34:02.501554 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:34:02.501556 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:34:02.501557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:34:02.501743 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:34:02.501821 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:02.502403 (+ 582us) server_negotiation.cc:299] Negotiation successful
0926 12:34:02.502538 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.2@KRBTEST.COM" created.
I20260926 12:34:02.503633 19041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:02.490660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.1:60449)
0926 12:34:02.491188 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:02.491252 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:02.492547 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:02.493116 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:02.493126 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:02.493498 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:02.494001 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:02.494021 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:02.494825 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:02.494829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:02.495841 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:02.495850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:02.496437 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:02.496440 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:02.496586 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:02.497259 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:02.497276 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:02.498920 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:34:02.500743 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:34:02.500748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:34:02.500760 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:34:02.501026 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:34:02.501319 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:34:02.501321 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:34:02.501323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:34:02.501466 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:34:02.501856 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:34:02.501863 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:34:02.502125 (+ 262us) client_negotiation.cc:769] Sending connection context
0926 12:34:02.502307 (+ 182us) client_negotiation.cc:240] Negotiation successful
0926 12:34:02.502617 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":153,"threads_started":1}
I20260926 12:34:02.505061 19039 heartbeater.cc:347] Connected to a master server at 127.10.98.62:34931
I20260926 12:34:02.505329 19039 heartbeater.cc:464] Registering TS with master...
I20260926 12:34:02.505825 19039 heartbeater.cc:511] Master 127.10.98.62:34931 requested a full tablet report, sending...
I20260926 12:34:02.507553 18848 ts_manager.cc:194] Registered new tserver with Master: 8ede2d7d28af463699a35965b0553b7f (127.10.98.1:42487)
I20260926 12:34:02.509212 18848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:60449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
I20260926 12:34:02.521951 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:34931
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:36457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:34:02.633868 19044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:34:02.634153 19044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:34:02.634196 19044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:34:02.638230 19044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:34:02.638289 19044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:34:02.638348 19044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:34:02.643656 19044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:36457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:34931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:34:02.644755 19044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:34:02.645812 19044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:34:02.655517 19050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:02.655532 19049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:02.655550 19052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:02.655643 19044 server_base.cc:1034] running on GCE node
I20260926 12:34:02.656370 19044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:34:02.657018 19044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:34:02.658174 19044 hybrid_clock.cc:648] HybridClock initialized: now 1790426042658153 us; error 35 us; skew 500 ppm
Sep 26 12:34:02 dist-test-slave-t7tk krb5kdc[18803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426042, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:34:02.660909 19044 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:34:02.661919 19044 webserver.cc:460] Webserver started at http://127.10.98.2:40295/ using document root <none> and password file <none>
I20260926 12:34:02.662461 19044 fs_manager.cc:359] Metadata directory not provided
I20260926 12:34:02.662528 19044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:34:02.662729 19044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:34:02.664747 19044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "0a1f51a5042c4ce79801dfc0f2e5a8d0"
format_stamp: "Formatted at 2026-09-26 12:34:02 on dist-test-slave-t7tk"
server_key: "3598a7f0705a39182f1b6870d8ef9055"
server_key_iv: "037d2067dc84e15b148c8fb887593a90"
server_key_version: "encryptionkey@0"
I20260926 12:34:02.665334 19044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "0a1f51a5042c4ce79801dfc0f2e5a8d0"
format_stamp: "Formatted at 2026-09-26 12:34:02 on dist-test-slave-t7tk"
server_key: "3598a7f0705a39182f1b6870d8ef9055"
server_key_iv: "037d2067dc84e15b148c8fb887593a90"
server_key_version: "encryptionkey@0"
I20260926 12:34:02.668977 19044 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:34:02.671456 19059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:02.672771 19044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:34:02.672890 19044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "0a1f51a5042c4ce79801dfc0f2e5a8d0"
format_stamp: "Formatted at 2026-09-26 12:34:02 on dist-test-slave-t7tk"
server_key: "3598a7f0705a39182f1b6870d8ef9055"
server_key_iv: "037d2067dc84e15b148c8fb887593a90"
server_key_version: "encryptionkey@0"
I20260926 12:34:02.672998 19044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:34:02.689858 19044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:34:02.693151 19044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:34:02.693338 19044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:34:02.693944 19044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:34:02.694780 19044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:34:02.694828 19044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:02.694882 19044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:34:02.694909 19044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:02.703334 19044 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:34031
I20260926 12:34:02.703390 19172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:34031 every 8 connection(s)
I20260926 12:34:02.704252 19044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:34:02.708556 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 19044
I20260926 12:34:02.708698 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:34:02.708983 10632 external_mini_cluster.cc:1428] Setting key 1fb28dda5a7013320531425af2c5ba7f
Sep 26 12:34:02 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426042, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.98.3@KRBTEST.COM; defaulting to no policy
I20260926 12:34:02.719156 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:02.706123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:57443 (local address 127.10.98.62:34931)
0926 12:34:02.706268 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:02.706273 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:02.707312 (+ 1039us) server_negotiation.cc:407] Connection header received
0926 12:34:02.708211 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:02.708214 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:02.708270 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:02.708345 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:02.709793 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:02.710401 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:02.711555 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:02.711780 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:02.711867 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:02.714901 (+ 3034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:34:02.714905 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:34:02.714908 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:34:02.714940 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:34:02.716604 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:34:02.717349 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:34:02.717353 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:34:02.717355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:34:02.717440 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:34:02.717823 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:34:02.717827 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:34:02.717828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:34:02.718026 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:34:02.718121 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:02.718910 (+ 789us) server_negotiation.cc:299] Negotiation successful
0926 12:34:02.719031 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:34:02.720166 19175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:02.706515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.2:57443)
0926 12:34:02.707158 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:02.707200 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:02.708007 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:02.708531 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:02.708538 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:02.709017 (+ 479us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:02.709642 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:02.709662 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:02.710524 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:02.710527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:02.711436 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:02.711446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:02.711921 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:02.711925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:02.712108 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:02.712748 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:02.712766 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:02.714646 (+ 1880us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:34:02.716773 (+ 2127us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:34:02.716779 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:34:02.716793 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:34:02.717157 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:34:02.717530 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:34:02.717532 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:34:02.717535 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:34:02.717723 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:34:02.718150 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:34:02.718157 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:34:02.718555 (+ 398us) client_negotiation.cc:769] Sending connection context
0926 12:34:02.718783 (+ 228us) client_negotiation.cc:240] Negotiation successful
0926 12:34:02.719162 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":112,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.3@KRBTEST.COM" created.
I20260926 12:34:02.721601 19173 heartbeater.cc:347] Connected to a master server at 127.10.98.62:34931
I20260926 12:34:02.721848 19173 heartbeater.cc:464] Registering TS with master...
I20260926 12:34:02.722525 19173 heartbeater.cc:511] Master 127.10.98.62:34931 requested a full tablet report, sending...
I20260926 12:34:02.723822 18848 ts_manager.cc:194] Registered new tserver with Master: 0a1f51a5042c4ce79801dfc0f2e5a8d0 (127.10.98.2:34031)
I20260926 12:34:02.724558 18848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:57443
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
I20260926 12:34:02.741397 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:34931
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:36457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:34:02.852780 19178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:34:02.853045 19178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:34:02.853089 19178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:34:02.857249 19178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:34:02.857319 19178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:34:02.857380 19178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:34:02.862846 19178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:36457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:34931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:34:02.863902 19178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:34:02.864878 19178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:34:02.874639 19184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:02.874663 19186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:02.874714 19183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:02.874873 19178 server_base.cc:1034] running on GCE node
I20260926 12:34:02.875412 19178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:34:02.876139 19178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:34:02.877341 19178 hybrid_clock.cc:648] HybridClock initialized: now 1790426042877276 us; error 35 us; skew 500 ppm
Sep 26 12:34:02 dist-test-slave-t7tk krb5kdc[18803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426042, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:34:02.880043 19178 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:34:02.880923 19178 webserver.cc:460] Webserver started at http://127.10.98.3:39067/ using document root <none> and password file <none>
I20260926 12:34:02.881500 19178 fs_manager.cc:359] Metadata directory not provided
I20260926 12:34:02.881560 19178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:34:02.881722 19178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:34:02.883692 19178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "ab5321d2edef4ea99624d22c5f05346f"
format_stamp: "Formatted at 2026-09-26 12:34:02 on dist-test-slave-t7tk"
server_key: "1eb205d85cb4f7428906a508a1a4427f"
server_key_iv: "042a80bc4fcfdcf8249baa3919ef47d0"
server_key_version: "encryptionkey@0"
I20260926 12:34:02.884207 19178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "ab5321d2edef4ea99624d22c5f05346f"
format_stamp: "Formatted at 2026-09-26 12:34:02 on dist-test-slave-t7tk"
server_key: "1eb205d85cb4f7428906a508a1a4427f"
server_key_iv: "042a80bc4fcfdcf8249baa3919ef47d0"
server_key_version: "encryptionkey@0"
I20260926 12:34:02.887962 19178 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 12:34:02.890477 19193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:02.891776 19178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:34:02.891881 19178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "ab5321d2edef4ea99624d22c5f05346f"
format_stamp: "Formatted at 2026-09-26 12:34:02 on dist-test-slave-t7tk"
server_key: "1eb205d85cb4f7428906a508a1a4427f"
server_key_iv: "042a80bc4fcfdcf8249baa3919ef47d0"
server_key_version: "encryptionkey@0"
I20260926 12:34:02.891971 19178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:34:02.903918 19178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:34:02.907006 19178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:34:02.907161 19178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:34:02.907749 19178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:34:02.908531 19178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:34:02.908576 19178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:02.908614 19178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:34:02.908636 19178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:02.917081 19178 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:35679
I20260926 12:34:02.917263 19306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:35679 every 8 connection(s)
I20260926 12:34:02.918074 19178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
Sep 26 12:34:02 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426042, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:34:02.927912 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 19178
I20260926 12:34:02.928041 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:34:02.928323 10632 external_mini_cluster.cc:1428] Setting key 34982ff2769edd68a32c8f228b8e6855
I20260926 12:34:02.931851 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:02.919965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:58167 (local address 127.10.98.62:34931)
0926 12:34:02.920097 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:02.920100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:02.920976 (+ 876us) server_negotiation.cc:407] Connection header received
0926 12:34:02.922332 (+ 1356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:02.922335 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:02.922386 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:02.922459 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:02.923600 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:02.924134 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:02.925395 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:02.925641 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:02.925749 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:02.928473 (+ 2724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:34:02.928476 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:34:02.928478 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:34:02.928504 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:34:02.929906 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:34:02.930476 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:34:02.930479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:34:02.930481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:34:02.930566 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:34:02.930875 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:34:02.930878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:34:02.930879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:34:02.931063 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:34:02.931135 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:02.931614 (+ 479us) server_negotiation.cc:299] Negotiation successful
0926 12:34:02.931732 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:34:02.932667 19309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:02.920333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.3:58167)
0926 12:34:02.920783 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:02.920850 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:02.922101 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:02.922599 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:02.922606 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:02.922957 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:02.923454 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:02.923468 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:02.924362 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:02.924365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:02.925253 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:02.925262 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:02.925907 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:02.925910 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:02.926069 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:02.926674 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:02.926692 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:02.928292 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:34:02.930013 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:34:02.930020 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:34:02.930035 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:34:02.930380 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:34:02.930637 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:34:02.930639 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:34:02.930640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:34:02.930796 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:34:02.931137 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:34:02.931142 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:34:02.931406 (+ 264us) client_negotiation.cc:769] Sending connection context
0926 12:34:02.931565 (+ 159us) client_negotiation.cc:240] Negotiation successful
0926 12:34:02.931850 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":118,"threads_started":1}
I20260926 12:34:02.933931 19307 heartbeater.cc:347] Connected to a master server at 127.10.98.62:34931
I20260926 12:34:02.934187 19307 heartbeater.cc:464] Registering TS with master...
I20260926 12:34:02.934754 19307 heartbeater.cc:511] Master 127.10.98.62:34931 requested a full tablet report, sending...
I20260926 12:34:02.935833 18848 ts_manager.cc:194] Registered new tserver with Master: ab5321d2edef4ea99624d22c5f05346f (127.10.98.3:35679)
I20260926 12:34:02.936463 18848 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:58167
I20260926 12:34:02.942310 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.10.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.1.keytab.
Entry for principal oryx/127.10.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.1.keytab.
WARNING: no policy specified for oryx/127.10.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.2.keytab.
Entry for principal oryx/127.10.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.2.keytab.
WARNING: no policy specified for oryx/127.10.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.10.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.10.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.3.keytab.
Entry for principal oryx/127.10.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.3.keytab.
I20260926 12:34:03.017192 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 18910
I20260926 12:34:03.022333 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 19044
I20260926 12:34:03.027652 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 19178
I20260926 12:34:03.032580 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 18816
I20260926 12:34:03.038257 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:34931
--webserver_interface=127.10.98.62
--webserver_port=37305
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:36457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:34931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:34:03.150929 19316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:34:03.151237 19316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:34:03.151295 19316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:34:03.155570 19316 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:34:03.155651 19316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:34:03.155679 19316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:34:03.155701 19316 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:34:03.155761 19316 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:34:03.161412 19316 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:36457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:34931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:34931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=37305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:34:03.162555 19316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:34:03.163666 19316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:34:03.172178 19322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:03.172178 19321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:03.172178 19324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:03.172518 19316 server_base.cc:1034] running on GCE node
I20260926 12:34:03.173177 19316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:34:03.174319 19316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:34:03.175498 19316 hybrid_clock.cc:648] HybridClock initialized: now 1790426043175464 us; error 49 us; skew 500 ppm
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426043, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:34:03.178503 19316 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:34:03.179561 19316 webserver.cc:460] Webserver started at http://127.10.98.62:37305/ using document root <none> and password file <none>
I20260926 12:34:03.180158 19316 fs_manager.cc:359] Metadata directory not provided
I20260926 12:34:03.180246 19316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:34:03.184489 19316 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:34:03.186879 19331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:03.188346 19316 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:34:03.188493 19316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "94feb09e1ad2492eaae6bca6a4de85b0"
format_stamp: "Formatted at 2026-09-26 12:34:02 on dist-test-slave-t7tk"
server_key: "0f677b1e10a35d397063636da691ef64"
server_key_iv: "ef3019e4fa7989553693adf9f19c0fe3"
server_key_version: "encryptionkey@0"
I20260926 12:34:03.189051 19316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:34:03.210752 19316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:34:03.214087 19316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:34:03.214258 19316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:34:03.221316 19316 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:34931
I20260926 12:34:03.221354 19383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:34931 every 8 connection(s)
I20260926 12:34:03.222452 19316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:34:03.224862 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 19316
I20260926 12:34:03.225386 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:42487
--local_ip_for_outbound_sockets=127.10.98.1
--tserver_master_addrs=127.10.98.62:34931
--webserver_port=33915
--webserver_interface=127.10.98.1
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:36457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.98.1
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:34:03.227499 19384 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:34:03.229271 19384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0: Bootstrap starting.
I20260926 12:34:03.245623 19384 log.cc:824] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0: Log is configured to *not* fsync() on all Append() calls
I20260926 12:34:03.252350 19384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:34:03.252698 19384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0: Bootstrap complete.
I20260926 12:34:03.256783 19384 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94feb09e1ad2492eaae6bca6a4de85b0" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 34931 } }
I20260926 12:34:03.257241 19384 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94feb09e1ad2492eaae6bca6a4de85b0, State: Initialized, Role: FOLLOWER
I20260926 12:34:03.257735 19384 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94feb09e1ad2492eaae6bca6a4de85b0" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 34931 } }
I20260926 12:34:03.257865 19384 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:34:03.257931 19384 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:34:03.258021 19384 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [term 1 FOLLOWER]: Advancing to term 2
I20260926 12:34:03.260164 19384 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94feb09e1ad2492eaae6bca6a4de85b0" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 34931 } }
I20260926 12:34:03.260535 19384 leader_election.cc:302] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94feb09e1ad2492eaae6bca6a4de85b0; no voters:
I20260926 12:34:03.260879 19384 leader_election.cc:288] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 12:34:03.261026 19388 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 12:34:03.261313 19388 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [term 2 LEADER]: Becoming Leader. State: Replica: 94feb09e1ad2492eaae6bca6a4de85b0, State: Running, Role: LEADER
I20260926 12:34:03.261657 19388 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94feb09e1ad2492eaae6bca6a4de85b0" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 34931 } }
I20260926 12:34:03.262151 19384 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:34:03.263211 19390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94feb09e1ad2492eaae6bca6a4de85b0. Latest consensus state: current_term: 2 leader_uuid: "94feb09e1ad2492eaae6bca6a4de85b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94feb09e1ad2492eaae6bca6a4de85b0" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 34931 } } }
I20260926 12:34:03.263355 19390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [sys.catalog]: This master's current role is: LEADER
I20260926 12:34:03.263581 19389 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "94feb09e1ad2492eaae6bca6a4de85b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94feb09e1ad2492eaae6bca6a4de85b0" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 34931 } } }
I20260926 12:34:03.263664 19389 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0 [sys.catalog]: This master's current role is: LEADER
I20260926 12:34:03.267220 19403 catalog_manager.cc:1259] Loaded cluster ID: 6014278bcfdb46e785bd7f279e260bad
I20260926 12:34:03.267313 19403 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0: loading cluster ID for follower catalog manager: success
I20260926 12:34:03.269747 19403 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0: acquiring CA information for follower catalog manager: success
I20260926 12:34:03.271356 19403 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 12:34:03.271483 19397 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:34:03.272684 19397 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:34:03.273118 19397 catalog_manager.cc:1259] Loaded cluster ID: 6014278bcfdb46e785bd7f279e260bad
I20260926 12:34:03.273182 19397 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:34:03.273828 19397 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:34:03.274286 19397 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 94feb09e1ad2492eaae6bca6a4de85b0: Loaded TSK: 0
I20260926 12:34:03.275031 19397 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 12:34:03.342546 19386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:34:03.342828 19386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:34:03.342871 19386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:34:03.348074 19386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:34:03.348147 19386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:34:03.348230 19386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:34:03.353909 19386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:36457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.1.keytab
--principal=oryx/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:42487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=33915
--tserver_master_addrs=127.10.98.62:34931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:34:03.354993 19386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:34:03.355997 19386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:34:03.366154 19413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:03.366158 19410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:03.366346 19386 server_base.cc:1034] running on GCE node
W20260926 12:34:03.366273 19411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:03.366918 19386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:34:03.367772 19386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:34:03.368978 19386 hybrid_clock.cc:648] HybridClock initialized: now 1790426043368951 us; error 38 us; skew 500 ppm
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426043, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:34:03.372581 19386 init.cc:377] Logged in from keytab as oryx/127.10.98.1@KRBTEST.COM (short username oryx)
I20260926 12:34:03.373553 19386 webserver.cc:460] Webserver started at http://127.10.98.1:33915/ using document root <none> and password file <none>
I20260926 12:34:03.374119 19386 fs_manager.cc:359] Metadata directory not provided
I20260926 12:34:03.374202 19386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:34:03.378422 19386 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:34:03.380900 19420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:03.382460 19386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:34:03.382583 19386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "8ede2d7d28af463699a35965b0553b7f"
format_stamp: "Formatted at 2026-09-26 12:34:02 on dist-test-slave-t7tk"
server_key: "6eb2eb97432c932040069aef34548755"
server_key_iv: "422e62a1e38c2420cd8e4e419cc30de6"
server_key_version: "encryptionkey@0"
I20260926 12:34:03.383191 19386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:34:03.395718 19386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:34:03.398891 19386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:34:03.399065 19386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:34:03.399655 19386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:34:03.400478 19386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:34:03.400524 19386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:03.400566 19386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:34:03.400594 19386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:03.408937 19386 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:42487
I20260926 12:34:03.408977 19533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:42487 every 8 connection(s)
I20260926 12:34:03.409981 19386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:34:03.412001 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 19386
I20260926 12:34:03.412482 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:34031
--local_ip_for_outbound_sockets=127.10.98.2
--tserver_master_addrs=127.10.98.62:34931
--webserver_port=40295
--webserver_interface=127.10.98.2
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:36457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.98.2
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 12:34:03.424715 19536 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:03.412003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:57231 (local address 127.10.98.62:34931)
0926 12:34:03.412445 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:03.412461 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:03.413021 (+ 560us) server_negotiation.cc:407] Connection header received
0926 12:34:03.414789 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:03.414800 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:03.415136 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:03.415451 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:03.416639 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.417672 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.418844 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.419346 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.419505 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:03.423940 (+ 4435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.1:57231: BlockingRecv error: recv got EOF from 127.10.98.1:57231 (error 108)
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":104,"threads_started":1}
I20260926 12:34:03.424724 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:03.412321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.1:57231)
0926 12:34:03.412849 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:03.412903 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:03.414337 (+ 1434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:03.415553 (+ 1216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:03.415561 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:03.415999 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:03.416473 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:03.416487 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.417802 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.417804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:03.418720 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:03.418730 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.419517 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.419521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:03.419713 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:03.420425 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:03.420451 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:03.423648 (+ 3197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":139,"threads_started":1}
W20260926 12:34:03.425324 19534 heartbeater.cc:644] Failed to heartbeat to 127.10.98.62:34931 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.98.62:34931: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:03.430810 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:03.425897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.1:51821)
0926 12:34:03.426022 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:03.426038 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:03.426131 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:03.426369 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:03.426371 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:03.426533 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:03.426633 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:03.426639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.427411 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.427414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:03.427968 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:03.427976 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.428399 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.428403 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:03.428492 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:03.428942 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:03.428954 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:03.430677 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 12:34:03.430939 19536 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:03.425976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:51821 (local address 127.10.98.62:34931)
0926 12:34:03.426079 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:03.426083 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:03.426098 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:34:03.426225 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:03.426228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:03.426273 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:03.426341 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:03.426778 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.427307 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.428078 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.428300 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.428387 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:03.430835 (+ 2448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.1:51821: BlockingRecv error: recv got EOF from 127.10.98.1:51821 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:03.435858 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:03.431345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.1:45701)
0926 12:34:03.431436 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:03.431449 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:03.431523 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:03.431753 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:03.431756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:03.431892 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:03.431965 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:03.431969 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.432651 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.432655 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:03.433194 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:03.433202 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.433607 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.433611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:03.433684 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:03.434098 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:03.434109 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:03.435733 (+ 1624us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 12:34:03.436004 19536 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:03.431445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:45701 (local address 127.10.98.62:34931)
0926 12:34:03.431570 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:03.431574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:03.431586 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:34:03.431622 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:03.431624 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:03.431660 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:03.431729 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:03.432066 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.432554 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.433309 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.433513 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.433596 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:03.435889 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.1:45701: BlockingRecv error: recv got EOF from 127.10.98.1:45701 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 12:34:03.436088 19534 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:34:03.527199 19538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:34:03.527494 19538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:34:03.527539 19538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:34:03.531738 19538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:34:03.531810 19538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:34:03.531870 19538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:34:03.537243 19538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:36457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.2.keytab
--principal=oryx/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:34031
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=40295
--tserver_master_addrs=127.10.98.62:34931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:34:03.538340 19538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:34:03.539389 19538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:34:03.549324 19543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:03.549307 19546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:03.549374 19544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:03.549706 19538 server_base.cc:1034] running on GCE node
I20260926 12:34:03.550113 19538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:34:03.550767 19538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:34:03.551898 19538 hybrid_clock.cc:648] HybridClock initialized: now 1790426043551859 us; error 51 us; skew 500 ppm
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426043, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:34:03.555482 19538 init.cc:377] Logged in from keytab as oryx/127.10.98.2@KRBTEST.COM (short username oryx)
I20260926 12:34:03.556749 19538 webserver.cc:460] Webserver started at http://127.10.98.2:40295/ using document root <none> and password file <none>
I20260926 12:34:03.557502 19538 fs_manager.cc:359] Metadata directory not provided
I20260926 12:34:03.557615 19538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:34:03.561764 19538 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 12:34:03.564209 19553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:03.565856 19538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:34:03.565981 19538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "0a1f51a5042c4ce79801dfc0f2e5a8d0"
format_stamp: "Formatted at 2026-09-26 12:34:02 on dist-test-slave-t7tk"
server_key: "3598a7f0705a39182f1b6870d8ef9055"
server_key_iv: "037d2067dc84e15b148c8fb887593a90"
server_key_version: "encryptionkey@0"
I20260926 12:34:03.566546 19538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:34:03.577395 19538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:34:03.580529 19538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:34:03.580703 19538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:34:03.581326 19538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:34:03.582132 19538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:34:03.582196 19538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:03.582273 19538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:34:03.582301 19538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:03.590860 19538 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:34031
I20260926 12:34:03.590890 19666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:34031 every 8 connection(s)
I20260926 12:34:03.591928 19538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:34:03.599143 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 19538
I20260926 12:34:03.599577 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:35679
--local_ip_for_outbound_sockets=127.10.98.3
--tserver_master_addrs=127.10.98.62:34931
--webserver_port=39067
--webserver_interface=127.10.98.3
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:36457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.10.98.3
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 12:34:03.604373 19536 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:03.593831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:43447 (local address 127.10.98.62:34931)
0926 12:34:03.593956 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:03.593960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:03.595081 (+ 1121us) server_negotiation.cc:407] Connection header received
0926 12:34:03.596618 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:03.596622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:03.596690 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:03.596764 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:03.598132 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.598670 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.599894 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.600113 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.600202 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:03.604262 (+ 4060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.2:43447: BlockingRecv error: recv got EOF from 127.10.98.2:43447 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:34:03.605413 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:03.594364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.2:43447)
0926 12:34:03.594907 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:03.594954 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:03.596356 (+ 1402us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:03.596975 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:03.596983 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:03.597443 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:03.597971 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:03.597985 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.598795 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.598798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:03.599759 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:03.599771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.600259 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.600264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:03.600442 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:03.601084 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:03.601132 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:03.604090 (+ 2958us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":132,"threads_started":1}
W20260926 12:34:03.605938 19667 heartbeater.cc:644] Failed to heartbeat to 127.10.98.62:34931 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.98.62:34931: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:03.612159 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:03.606442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.2:52963)
0926 12:34:03.606634 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:03.606649 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:03.606719 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:03.607030 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:03.607033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:03.607247 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:03.607350 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:03.607355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.608434 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.608437 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:03.609036 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:03.609045 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.609664 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.609667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:03.609743 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:03.610211 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:03.610224 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:03.612055 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124}
W20260926 12:34:03.612344 19536 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:03.606573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:52963 (local address 127.10.98.62:34931)
0926 12:34:03.606753 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:03.606758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:03.606776 (+ 18us) server_negotiation.cc:407] Connection header received
0926 12:34:03.606830 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:03.606832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:03.606882 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:03.606998 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:03.607475 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.608301 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.609191 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.609523 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.609651 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:03.612223 (+ 2572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.2:52963: BlockingRecv error: recv got EOF from 127.10.98.2:52963 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:03.618263 19669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:03.612848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.2:42123)
0926 12:34:03.613009 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:03.613021 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:03.613123 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:03.613349 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:03.613352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:03.613527 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:03.613633 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:03.613639 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.614331 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.614334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:03.615198 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:03.615207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.615591 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.615594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:03.615681 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:03.616143 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:03.616157 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:03.618040 (+ 1883us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99}
W20260926 12:34:03.618297 19536 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:03.612925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:42123 (local address 127.10.98.62:34931)
0926 12:34:03.613025 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:03.613028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:03.613059 (+ 31us) server_negotiation.cc:407] Connection header received
0926 12:34:03.613218 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:03.613220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:03.613259 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:03.613320 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:03.613737 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.614238 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.615297 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.615500 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.615573 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:03.618191 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.2:42123: BlockingRecv error: recv got EOF from 127.10.98.2:42123 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 12:34:03.618585 19667 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:34:03.714139 19670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:34:03.714473 19670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:34:03.714534 19670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:34:03.718703 19670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:34:03.718770 19670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:34:03.718855 19670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:34:03.724335 19670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:36457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/krb5kdc/oryx_127.10.98.3.keytab
--principal=oryx/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:35679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=39067
--tserver_master_addrs=127.10.98.62:34931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:34:03.725497 19670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:34:03.726516 19670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:34:03.736346 19675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:03.736346 19678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:03.736358 19676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:03.736639 19670 server_base.cc:1034] running on GCE node
I20260926 12:34:03.737183 19670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:34:03.737861 19670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:34:03.739044 19670 hybrid_clock.cc:648] HybridClock initialized: now 1790426043739023 us; error 36 us; skew 500 ppm
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426043, etypes {rep=17 tkt=17 ses=17}, oryx/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:34:03.741972 19670 init.cc:377] Logged in from keytab as oryx/127.10.98.3@KRBTEST.COM (short username oryx)
I20260926 12:34:03.742916 19670 webserver.cc:460] Webserver started at http://127.10.98.3:39067/ using document root <none> and password file <none>
I20260926 12:34:03.743472 19670 fs_manager.cc:359] Metadata directory not provided
I20260926 12:34:03.743561 19670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:34:03.747997 19670 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:34:03.750454 19685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:03.751886 19670 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:34:03.752022 19670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "ab5321d2edef4ea99624d22c5f05346f"
format_stamp: "Formatted at 2026-09-26 12:34:02 on dist-test-slave-t7tk"
server_key: "1eb205d85cb4f7428906a508a1a4427f"
server_key_iv: "042a80bc4fcfdcf8249baa3919ef47d0"
server_key_version: "encryptionkey@0"
I20260926 12:34:03.752650 19670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:34:03.766283 19670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:34:03.769845 19670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:34:03.770042 19670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:34:03.770656 19670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:34:03.771513 19670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:34:03.771561 19670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:03.771615 19670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:34:03.771648 19670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:03.780386 19670 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:35679
I20260926 12:34:03.780411 19798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:35679 every 8 connection(s)
I20260926 12:34:03.781498 19670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:34:03.786536 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 19670
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 12:34:03.793529 19536 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:03.783408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:37005 (local address 127.10.98.62:34931)
0926 12:34:03.783536 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:03.783540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:03.784520 (+ 980us) server_negotiation.cc:407] Connection header received
0926 12:34:03.785850 (+ 1330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:03.785854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:03.785911 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:03.785995 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:03.787396 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.788110 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.789652 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.789986 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.790101 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:03.793426 (+ 3325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.3:37005: BlockingRecv error: recv got EOF from 127.10.98.3:37005 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:34:03.794389 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:03.783828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.3:37005)
0926 12:34:03.784356 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:03.784401 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:03.785614 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:03.786182 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:03.786189 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:03.786551 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:03.787189 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:03.787206 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.788242 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.788245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:03.789528 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:03.789538 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.790159 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.790163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:03.790329 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:03.790883 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:03.790903 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:03.793321 (+ 2418us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
W20260926 12:34:03.794893 19799 heartbeater.cc:644] Failed to heartbeat to 127.10.98.62:34931 (0 consecutive failures): Not authorized: Failed to ping master at 127.10.98.62:34931: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
I20260926 12:34:03.796772 19803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:03.787575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60602 (local address 127.10.98.62:34931)
0926 12:34:03.787789 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:03.787792 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:03.787803 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:34:03.787837 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:03.787840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:03.787886 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:03.787971 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:03.788715 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.789511 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.790625 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.790904 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.791010 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:03.792338 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:34:03.792344 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:34:03.792359 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:34:03.792385 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:34:03.794440 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:34:03.794972 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:34:03.794978 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:34:03.794984 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:34:03.795119 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:34:03.795565 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:34:03.795568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:34:03.795569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:34:03.795956 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:34:03.796135 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:03.796417 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 12:34:03.796645 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":75,"threads_started":1}
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:03.801026 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:03.795323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.3:52841)
0926 12:34:03.795488 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:03.795501 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:03.795567 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:03.795790 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:03.795793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:03.795946 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:03.796041 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:03.796046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.796791 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.796794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:03.797609 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:03.797619 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.798144 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.798146 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:03.798223 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:03.798641 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:03.798655 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:03.800937 (+ 2282us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101}
W20260926 12:34:03.801239 19536 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:03.795411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:52841 (local address 127.10.98.62:34931)
0926 12:34:03.795513 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:03.795516 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:03.795530 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:34:03.795652 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:03.795655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:03.795703 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:03.795764 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:03.796140 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.796701 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.797745 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.798040 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.798145 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:03.801080 (+ 2935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.3:52841: BlockingRecv error: recv got EOF from 127.10.98.3:52841 (error 108)
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":27}
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:03 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:03.806100 19801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:03.801541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.3:37291)
0926 12:34:03.801684 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:03.801700 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:03.801776 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:03.801971 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:03.801973 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:03.802092 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:03.802160 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:03.802164 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.802895 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.802904 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:03.803433 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:03.803441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.803877 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.803880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:03.803944 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:03.804360 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:03.804371 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:03.806005 (+ 1634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 12:34:03.806262 19536 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:03.801639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:37291 (local address 127.10.98.62:34931)
0926 12:34:03.801801 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:03.801804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:03.801817 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:34:03.801847 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:03.801849 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:03.801883 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:03.801950 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:03.802288 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.802796 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.803545 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:03.803758 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:03.803843 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:03.806147 (+ 2304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.3:37291: BlockingRecv error: recv got EOF from 127.10.98.3:37291 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260926 12:34:03.806409 19799 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 12:34:04 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:04 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:04.442262 19804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:04.436756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.1:47641)
0926 12:34:04.437067 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:04.437082 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:04.437185 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:04.437433 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:04.437435 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:04.437606 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:04.437726 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:04.437732 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:04.438530 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:04.438533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:04.439125 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:04.439134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:04.439587 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:04.439589 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:04.439662 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:04.440152 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:04.440165 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:04.442136 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":98,"threads_started":1}
W20260926 12:34:04.442404 19805 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:04.436908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:47641 (local address 127.10.98.62:34931)
0926 12:34:04.437187 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:04.437191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:04.437204 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:34:04.437245 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:04.437248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:04.437298 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:04.437378 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:04.437870 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:04.438424 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:04.439236 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:04.439482 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:04.439562 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:04.442274 (+ 2712us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.1:47641: BlockingRecv error: recv got EOF from 127.10.98.1:47641 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":99,"threads_started":1}
Sep 26 12:34:04 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:04 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:04.624631 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:04.619199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.2:34375)
0926 12:34:04.619486 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:04.619501 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:04.619621 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:04.619877 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:04.619880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:04.620081 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:04.620176 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:04.620183 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:04.620935 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:04.620938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:04.621557 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:04.621566 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:04.621986 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:04.621990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:04.622067 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:04.622534 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:04.622545 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:04.624505 (+ 1960us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":81,"threads_started":1}
W20260926 12:34:04.624773 19805 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:04.619290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:34375 (local address 127.10.98.62:34931)
0926 12:34:04.619488 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:04.619492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:04.619578 (+ 86us) server_negotiation.cc:407] Connection header received
0926 12:34:04.619713 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:04.619715 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:04.619768 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:04.619838 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:04.620299 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:04.620824 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:04.621671 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:04.621887 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:04.621973 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:04.624650 (+ 2677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.2:34375: BlockingRecv error: recv got EOF from 127.10.98.2:34375 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Sep 26 12:34:04 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:04 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:04.812233 19807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:04.806986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.3:46107)
0926 12:34:04.807272 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:04.807285 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:04.807387 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:04.807602 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:04.807604 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:04.807794 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:04.807894 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:04.807900 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:04.808694 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:04.808697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:04.809367 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:04.809378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:04.809787 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:04.809790 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:04.809854 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:04.810338 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:04.810350 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:04.812099 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":83,"threads_started":1}
W20260926 12:34:04.812451 19805 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:04.807071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:46107 (local address 127.10.98.62:34931)
0926 12:34:04.807204 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:04.807208 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:04.807353 (+ 145us) server_negotiation.cc:407] Connection header received
0926 12:34:04.807461 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:04.807463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:04.807508 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:04.807570 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:04.807992 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:04.808545 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:04.809473 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:04.809685 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:04.809758 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:04.812319 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.3:46107: BlockingRecv error: recv got EOF from 127.10.98.3:46107 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 12:34:05 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:05 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:05.449409 19808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:05.443311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.1:51987)
0926 12:34:05.443642 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:05.443657 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:05.443766 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:05.444134 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:05.444137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:05.444411 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:05.444524 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:05.444531 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:05.445375 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:05.445377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:05.445967 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:05.445976 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:05.446542 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:05.446545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:05.446615 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:05.447146 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:05.447158 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:05.449263 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":122,"threads_started":1}
W20260926 12:34:05.449534 19809 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:05.443390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:51987 (local address 127.10.98.62:34931)
0926 12:34:05.443654 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:05.443658 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:05.443733 (+ 75us) server_negotiation.cc:407] Connection header received
0926 12:34:05.443886 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:05.443889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:05.443947 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:05.444062 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:05.444687 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:05.445242 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:05.446116 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:05.446419 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:05.446521 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:05.449417 (+ 2896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.1:51987: BlockingRecv error: recv got EOF from 127.10.98.1:51987 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":63,"threads_started":1}
Sep 26 12:34:05 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:05 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:05.631222 19810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:05.625523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.2:52539)
0926 12:34:05.625827 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:05.625842 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:05.625942 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:05.626265 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:05.626268 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:05.626444 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:05.626541 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:05.626547 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:05.627329 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:05.627333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:05.627899 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:05.627907 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:05.628302 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:05.628304 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:05.628371 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:05.628837 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:05.628856 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:05.631075 (+ 2219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":77,"threads_started":1}
W20260926 12:34:05.631312 19809 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:05.625699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:52539 (local address 127.10.98.62:34931)
0926 12:34:05.625872 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:05.625876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:05.625894 (+ 18us) server_negotiation.cc:407] Connection header received
0926 12:34:05.626068 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:05.626071 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:05.626119 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:05.626203 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:05.626668 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:05.627217 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:05.627993 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:05.628198 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:05.628266 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:05.631213 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.2:52539: BlockingRecv error: recv got EOF from 127.10.98.2:52539 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 12:34:05 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:05 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:05.818717 19811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:05.813153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.3:39247)
0926 12:34:05.813453 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:05.813466 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:05.813561 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:05.813847 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:05.813873 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:05.814054 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:05.814151 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:05.814157 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:05.814962 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:05.814965 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:05.815552 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:05.815562 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:05.816047 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:05.816050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:05.816115 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:05.816661 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:05.816673 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:05.818523 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":69,"threads_started":1}
W20260926 12:34:05.818837 19809 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:05.813387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:39247 (local address 127.10.98.62:34931)
0926 12:34:05.813510 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:05.813514 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:05.813528 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:34:05.813640 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:05.813642 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:05.813686 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:05.813781 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:05.814290 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:05.814830 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:05.815682 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:05.815930 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:05.816015 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:05.818682 (+ 2667us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.3:39247: BlockingRecv error: recv got EOF from 127.10.98.3:39247 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 12:34:06 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:06 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:06.463794 19812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:06.450261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.1:55269)
0926 12:34:06.450561 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:06.450575 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:06.450679 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:06.450990 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:06.450993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:06.451254 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:06.451363 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:06.451370 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:06.452147 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:06.452149 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:06.452825 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:06.452836 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:06.453326 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:06.453329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:06.453401 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:06.461283 (+ 7882us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:06.461299 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:06.463666 (+ 2367us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":89,"threads_started":1}
W20260926 12:34:06.464000 19813 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:06.450405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:55269 (local address 127.10.98.62:34931)
0926 12:34:06.450621 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:06.450625 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:06.450640 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:34:06.450786 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:06.450789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:06.450832 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:06.450917 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:06.451496 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:06.452030 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:06.452948 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:06.453199 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:06.453277 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:06.463843 (+ 10566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.1:55269: BlockingRecv error: recv got EOF from 127.10.98.1:55269 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":61,"threads_started":1}
Sep 26 12:34:06 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:06 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:06.638345 19814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:06.632098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.2:48539)
0926 12:34:06.632443 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:06.632461 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:06.632596 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:06.632850 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:06.632852 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:06.633070 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:06.633220 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:06.633228 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:06.634249 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:06.634253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:06.635074 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:06.635086 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:06.635604 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:06.635607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:06.635708 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:06.636210 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:06.636222 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:06.638195 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":91,"threads_started":1}
W20260926 12:34:06.638470 19813 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:06.632242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:48539 (local address 127.10.98.62:34931)
0926 12:34:06.632458 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:06.632463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:06.632545 (+ 82us) server_negotiation.cc:407] Connection header received
0926 12:34:06.632693 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:06.632696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:06.632744 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:06.632812 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:06.633366 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:06.634078 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:06.635207 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:06.635465 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:06.635554 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:06.638361 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.2:48539: BlockingRecv error: recv got EOF from 127.10.98.2:48539 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 26 12:34:06 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:06 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:06.825356 19815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:06.819607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.3:59017)
0926 12:34:06.819934 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:06.819950 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:06.820069 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:06.820352 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:06.820355 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:06.820558 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:06.820655 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:06.820661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:06.821512 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:06.821516 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:06.822092 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:06.822101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:06.822545 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:06.822548 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:06.822629 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:06.823140 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:06.823160 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:06.825177 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":104,"threads_started":1}
W20260926 12:34:06.825630 19813 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:06.819771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:59017 (local address 127.10.98.62:34931)
0926 12:34:06.819934 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:06.819939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:06.820023 (+ 84us) server_negotiation.cc:407] Connection header received
0926 12:34:06.820157 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:06.820161 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:06.820206 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:06.820319 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:06.820769 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:06.821416 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:06.822205 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:06.822439 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:06.822529 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:06.825393 (+ 2864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.3:59017: BlockingRecv error: recv got EOF from 127.10.98.3:59017 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 26 12:34:07 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:07 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:07.470333 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:07.464612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.1:33277)
0926 12:34:07.464920 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:07.464935 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:07.465057 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:07.465456 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:07.465460 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:07.465642 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:07.465746 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:07.465752 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:07.466553 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:07.466556 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:07.467133 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:07.467141 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:07.467659 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:07.467662 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:07.467720 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:07.468219 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:07.468231 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:07.470154 (+ 1923us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":76,"threads_started":1}
W20260926 12:34:07.470472 19817 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:07.464651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:33277 (local address 127.10.98.62:34931)
0926 12:34:07.464900 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:07.464905 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:07.465008 (+ 103us) server_negotiation.cc:407] Connection header received
0926 12:34:07.465186 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:07.465189 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:07.465246 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:07.465391 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:07.465855 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:07.466447 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:07.467276 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:07.467564 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:07.467650 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:07.470301 (+ 2651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.1:33277: BlockingRecv error: recv got EOF from 127.10.98.1:33277 (error 108)
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":59,"threads_started":1}
Sep 26 12:34:07 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:07 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:07.644946 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:07.639176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.2:33393)
0926 12:34:07.639480 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:07.639495 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:07.639621 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:07.639891 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:07.639894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:07.640094 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:07.640209 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:07.640216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:07.641012 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:07.641016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:07.641681 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:07.641694 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:07.642143 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:07.642146 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:07.642228 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:07.642828 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:07.642857 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:07.644796 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":88,"threads_started":1}
W20260926 12:34:07.645053 19817 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:07.639288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:33393 (local address 127.10.98.62:34931)
0926 12:34:07.639465 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:07.639470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:07.639571 (+ 101us) server_negotiation.cc:407] Connection header received
0926 12:34:07.639718 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:07.639721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:07.639774 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:07.639845 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:07.640329 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:07.640884 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:07.641800 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:07.642032 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:07.642120 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:07.644953 (+ 2833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.2:33393: BlockingRecv error: recv got EOF from 127.10.98.2:33393 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 12:34:07 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:07 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:07.831445 19819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:07.826114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.3:44025)
0926 12:34:07.826470 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:07.826484 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:07.826591 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:07.826798 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:07.826800 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:07.826975 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:07.827067 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:07.827072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:07.827788 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:07.827790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:07.828391 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:07.828399 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:07.828786 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:07.828788 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:07.828846 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:07.829417 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:07.829432 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:07.831278 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":163,"threads_started":1}
W20260926 12:34:07.831507 19817 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:07.826256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:44025 (local address 127.10.98.62:34931)
0926 12:34:07.826406 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:07.826410 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:07.826583 (+ 173us) server_negotiation.cc:407] Connection header received
0926 12:34:07.826657 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:07.826659 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:07.826705 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:07.826770 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:07.827170 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:07.827704 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:07.828490 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:07.828704 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:07.828778 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:07.831412 (+ 2634us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.3:44025: BlockingRecv error: recv got EOF from 127.10.98.3:44025 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 12:34:08 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:08 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:08.476729 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:08.471140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.1:47673)
0926 12:34:08.471430 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:08.471447 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:08.471546 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:08.471816 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:08.471818 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:08.471995 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:08.472089 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:08.472094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:08.472918 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:08.472921 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:08.473574 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:08.473584 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:08.474057 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:08.474061 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:08.474133 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:08.474722 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:08.474734 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:08.476597 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":94,"threads_started":1}
W20260926 12:34:08.476918 19821 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:08.471270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:47673 (local address 127.10.98.62:34931)
0926 12:34:08.471581 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:08.471585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:08.471599 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:34:08.471645 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:08.471647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:08.471687 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:08.471770 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:08.472194 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:08.472791 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:08.473691 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:08.473927 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:08.474011 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:08.476771 (+ 2760us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.1:47673: BlockingRecv error: recv got EOF from 127.10.98.1:47673 (error 108)
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":104,"threads_started":1}
Sep 26 12:34:08 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:08 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:08.651173 19822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:08.645799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.2:56287)
0926 12:34:08.646105 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:08.646118 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:08.646214 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:08.646490 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:08.646492 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:08.646673 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:08.646766 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:08.646771 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:08.647519 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:08.647522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:08.648130 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:08.648139 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:08.648567 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:08.648569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:08.648633 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:08.649171 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:08.649183 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:08.651022 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":103,"threads_started":1}
W20260926 12:34:08.651284 19821 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:08.645956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:56287 (local address 127.10.98.62:34931)
0926 12:34:08.646137 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:08.646140 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:08.646167 (+ 27us) server_negotiation.cc:407] Connection header received
0926 12:34:08.646296 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:08.646298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:08.646342 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:08.646421 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:08.646895 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:08.647414 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:08.648238 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:08.648467 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:08.648544 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:08.651184 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.2:56287: BlockingRecv error: recv got EOF from 127.10.98.2:56287 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 12:34:08 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:08 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:08.838497 19823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:08.832499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.3:56807)
0926 12:34:08.832799 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:08.832813 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:08.832915 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:08.833177 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:08.833180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:08.833386 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:08.833485 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:08.833491 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:08.834343 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:08.834346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:08.835084 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:08.835095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:08.835713 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:08.835716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:08.835802 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:08.836364 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:08.836378 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:08.838319 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":97,"threads_started":1}
W20260926 12:34:08.838690 19821 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:08.832618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:56807 (local address 127.10.98.62:34931)
0926 12:34:08.832776 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:08.832779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:08.832887 (+ 108us) server_negotiation.cc:407] Connection header received
0926 12:34:08.832992 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:08.832994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:08.833040 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:08.833151 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:08.833622 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:08.834223 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:08.835205 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:08.835566 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:08.835658 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:08.838574 (+ 2916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.3:56807: BlockingRecv error: recv got EOF from 127.10.98.3:56807 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 26 12:34:09 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:09 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:09.484006 19824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:09.477694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.1:57767)
0926 12:34:09.478027 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:09.478041 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:09.478135 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:09.478475 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:09.478477 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:09.478660 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:09.478768 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:09.478774 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:09.479680 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:09.479683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:09.480370 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:09.480387 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:09.480877 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:09.480882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:09.480962 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:09.481611 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:09.481645 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:09.483880 (+ 2235us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":127,"threads_started":1}
W20260926 12:34:09.484220 19825 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:09.477849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:57767 (local address 127.10.98.62:34931)
0926 12:34:09.478109 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:09.478113 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:09.478128 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:34:09.478222 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:09.478224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:09.478268 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:09.478415 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:09.478919 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:09.479540 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:09.480529 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:09.480758 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:09.480844 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:09.484069 (+ 3225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.1:57767: BlockingRecv error: recv got EOF from 127.10.98.1:57767 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":65,"threads_started":1}
Sep 26 12:34:09 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:09 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:09.658166 19826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:09.652112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.2:57995)
0926 12:34:09.652486 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:09.652499 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:09.652602 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:09.652878 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:09.652881 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:09.653060 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:09.653212 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:09.653218 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:09.654114 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:09.654116 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:09.654771 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:09.654780 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:09.655234 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:09.655237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:09.655301 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:09.655954 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:09.655974 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:09.658012 (+ 2038us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":82,"threads_started":1}
W20260926 12:34:09.658298 19825 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:09.652239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:57995 (local address 127.10.98.62:34931)
0926 12:34:09.652389 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:09.652394 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:09.652594 (+ 200us) server_negotiation.cc:407] Connection header received
0926 12:34:09.652678 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:09.652681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:09.652731 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:09.652807 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:09.653362 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:09.653989 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:09.654890 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:09.655115 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:09.655207 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:09.658193 (+ 2986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.2:57995: BlockingRecv error: recv got EOF from 127.10.98.2:57995 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 12:34:09 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:09 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:09.845000 19827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:09.839435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.3:54335)
0926 12:34:09.839783 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:09.839800 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:09.839929 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:09.840252 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:09.840256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:09.840491 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:09.840610 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:09.840617 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:09.841449 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:09.841454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:09.842083 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:09.842092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:09.842524 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:09.842527 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:09.842607 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:09.843078 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:09.843089 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:09.844875 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":113,"threads_started":1}
W20260926 12:34:09.845175 19825 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:09.839617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:54335 (local address 127.10.98.62:34931)
0926 12:34:09.839765 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:09.839769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:09.839997 (+ 228us) server_negotiation.cc:407] Connection header received
0926 12:34:09.840062 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:09.840065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:09.840123 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:09.840213 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:09.840722 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:09.841334 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:09.842198 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:09.842417 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:09.842510 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:09.845031 (+ 2521us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.3:54335: BlockingRecv error: recv got EOF from 127.10.98.3:54335 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 12:34:10 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:10 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:10.491804 19828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:10.485026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.1:52779)
0926 12:34:10.485487 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:10.485506 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:10.485618 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:10.485974 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:10.485995 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:10.486218 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:10.486324 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:10.486331 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:10.487396 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:10.487399 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:10.488029 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:10.488043 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:10.488662 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:10.488665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:10.488743 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:10.489356 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:10.489370 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:10.491676 (+ 2306us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":134,"threads_started":1}
W20260926 12:34:10.492020 19829 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:10.485210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:52779 (local address 127.10.98.62:34931)
0926 12:34:10.485521 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:10.485524 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:10.485564 (+ 40us) server_negotiation.cc:407] Connection header received
0926 12:34:10.485748 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:10.485750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:10.485802 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:10.485890 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:10.486510 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:10.487280 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:10.488183 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:10.488516 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:10.488610 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:10.491867 (+ 3257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.1:52779: BlockingRecv error: recv got EOF from 127.10.98.1:52779 (error 108)
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":57,"threads_started":1}
Sep 26 12:34:10 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:10 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:10.665409 19830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:10.659105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.2:33031)
0926 12:34:10.659433 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:10.659449 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:10.659569 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:10.659823 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:10.659825 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:10.660009 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:10.660103 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:10.660111 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:10.660865 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:10.660869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:10.661571 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:10.661582 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:10.662076 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:10.662080 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:10.662164 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:10.662998 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:10.663011 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:10.665243 (+ 2232us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":96,"threads_started":1}
W20260926 12:34:10.665566 19829 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:10.659215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:33031 (local address 127.10.98.62:34931)
0926 12:34:10.659377 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:10.659381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:10.659512 (+ 131us) server_negotiation.cc:407] Connection header received
0926 12:34:10.659656 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:10.659658 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:10.659701 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:10.659773 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:10.660226 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:10.660752 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:10.661699 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:10.661961 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:10.662057 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:10.665453 (+ 3396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.2:33031: BlockingRecv error: recv got EOF from 127.10.98.2:33031 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 26 12:34:10 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:10 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:10.851850 19831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:10.845991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.3:58867)
0926 12:34:10.846314 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:10.846329 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:10.846460 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:10.846746 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:10.846748 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:10.846940 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:10.847051 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:10.847057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:10.847883 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:10.847886 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:10.848498 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:10.848511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:10.849047 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:10.849051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:10.849164 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:10.849717 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:10.849733 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:10.851730 (+ 1997us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":86,"threads_started":1}
W20260926 12:34:10.852039 19829 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:10.846138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:58867 (local address 127.10.98.62:34931)
0926 12:34:10.846327 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:10.846330 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:10.846426 (+ 96us) server_negotiation.cc:407] Connection header received
0926 12:34:10.846546 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:10.846549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:10.846598 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:10.846672 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:10.847203 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:10.847751 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:10.848656 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:10.848917 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:10.849011 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:10.851913 (+ 2902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.3:58867: BlockingRecv error: recv got EOF from 127.10.98.3:58867 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 26 12:34:11 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:11 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:11.505038 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:11.492638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.1:40603)
0926 12:34:11.492955 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:11.492970 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:11.493128 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:11.493445 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:11.493448 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:11.493621 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:11.493740 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:11.493746 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:11.494539 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:11.494542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:11.495123 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:11.495133 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:11.495648 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:11.495651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:11.495728 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:11.502915 (+ 7187us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:11.502928 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:11.504902 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":81,"threads_started":1}
W20260926 12:34:11.505195 19833 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:11.492805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:40603 (local address 127.10.98.62:34931)
0926 12:34:11.493050 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:11.493054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:11.493070 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:34:11.493256 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:11.493259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:11.493309 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:11.493395 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:11.493847 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:11.494403 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:11.495255 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:11.495497 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:11.495587 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:11.505044 (+ 9457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.1:40603: BlockingRecv error: recv got EOF from 127.10.98.1:40603 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":61,"threads_started":1}
Sep 26 12:34:11 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:11 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:11.672379 19834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:11.666337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.2:60911)
0926 12:34:11.666679 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:11.666696 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:11.666826 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:11.667080 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:11.667083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:11.667289 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:11.667392 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:11.667398 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:11.668412 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:11.668415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:11.669142 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:11.669155 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:11.669661 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:11.669683 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:11.669784 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:11.670276 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:11.670288 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:11.672217 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":86,"threads_started":1}
W20260926 12:34:11.672497 19833 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:11.666513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:60911 (local address 127.10.98.62:34931)
0926 12:34:11.666690 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:11.666695 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:11.666777 (+ 82us) server_negotiation.cc:407] Connection header received
0926 12:34:11.666920 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:11.666923 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:11.666972 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:11.667044 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:11.667537 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:11.668223 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:11.669249 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:11.669535 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:11.669631 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:11.672372 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.2:60911: BlockingRecv error: recv got EOF from 127.10.98.2:60911 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 12:34:11 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:11 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:11.859052 19835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:11.852781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.3:42901)
0926 12:34:11.853089 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:11.853126 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:11.853238 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:11.853672 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:11.853675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:11.853872 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:11.853985 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:11.853990 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:11.854810 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:11.854813 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:11.855468 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:11.855480 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:11.856062 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:11.856065 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:11.856183 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:11.856824 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:11.856840 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:11.858824 (+ 1984us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":94,"threads_started":1}
W20260926 12:34:11.859150 19833 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:11.852930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:42901 (local address 127.10.98.62:34931)
0926 12:34:11.853063 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:11.853067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:11.853229 (+ 162us) server_negotiation.cc:407] Connection header received
0926 12:34:11.853454 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:11.853457 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:11.853510 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:11.853599 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:11.854137 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:11.854680 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:11.855649 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:11.855896 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:11.856007 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:11.859035 (+ 3028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.3:42901: BlockingRecv error: recv got EOF from 127.10.98.3:42901 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 12:34:12 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:12 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:12.511755 19836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:12.505979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.1:53023)
0926 12:34:12.506262 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:12.506275 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:12.506372 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:12.506682 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:12.506685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:12.506874 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:12.506974 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:12.506980 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:12.507789 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:12.507792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:12.508412 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:12.508423 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:12.508998 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:12.509002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:12.509117 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:12.509582 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:12.509594 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:12.511629 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":88,"threads_started":1}
W20260926 12:34:12.511952 19837 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:12.506143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:53023 (local address 127.10.98.62:34931)
0926 12:34:12.506378 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:12.506382 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:12.506410 (+ 28us) server_negotiation.cc:407] Connection header received
0926 12:34:12.506457 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:12.506460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:12.506503 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:12.506582 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:12.507112 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:12.507672 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:12.508539 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:12.508844 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:12.508961 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:12.511823 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.1:53023: BlockingRecv error: recv got EOF from 127.10.98.1:53023 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":86,"threads_started":1}
Sep 26 12:34:12 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:12 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:12.678988 19838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:12.673275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.2:40073)
0926 12:34:12.673556 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:12.673573 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:12.673698 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:12.673949 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:12.673952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:12.674149 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:12.674245 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:12.674251 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:12.675024 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:12.675028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:12.675598 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:12.675606 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:12.676037 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:12.676041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:12.676118 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:12.676614 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:12.676626 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:12.678840 (+ 2214us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
W20260926 12:34:12.679133 19837 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:12.673401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:40073 (local address 127.10.98.62:34931)
0926 12:34:12.673558 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:12.673563 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:12.673649 (+ 86us) server_negotiation.cc:407] Connection header received
0926 12:34:12.673791 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:12.673794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:12.673840 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:12.673910 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:12.674364 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:12.674910 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:12.675711 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:12.675934 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:12.676030 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:12.678996 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.2:40073: BlockingRecv error: recv got EOF from 127.10.98.2:40073 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 12:34:12 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:12 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:12.865787 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:12.860261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.3:40279)
0926 12:34:12.860618 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:12.860635 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:12.860752 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:12.861011 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:12.861014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:12.861257 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:12.861356 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:12.861362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:12.862120 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:12.862124 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:12.862741 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:12.862751 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:12.863210 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:12.863213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:12.863300 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:12.863815 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:12.863827 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:12.865670 (+ 1843us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":177,"threads_started":1}
W20260926 12:34:12.865990 19837 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:12.860352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:40279 (local address 127.10.98.62:34931)
0926 12:34:12.860479 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:12.860484 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:12.860717 (+ 233us) server_negotiation.cc:407] Connection header received
0926 12:34:12.860857 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:12.860860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:12.860906 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:12.860974 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:12.861469 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:12.862001 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:12.862848 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:12.863077 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:12.863175 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:12.865829 (+ 2654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.3:40279: BlockingRecv error: recv got EOF from 127.10.98.3:40279 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 12:34:13 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:13 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:13.518471 19840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:13.512641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.1:55165)
0926 12:34:13.513010 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:13.513025 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:13.513157 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:13.513400 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:13.513404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:13.513629 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:13.513764 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:13.513772 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:13.514686 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:13.514689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:13.515339 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:13.515352 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:13.515796 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:13.515799 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:13.515892 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:13.516357 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:13.516372 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:13.518345 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":98,"threads_started":1}
W20260926 12:34:13.518705 19841 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:13.512771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:55165 (local address 127.10.98.62:34931)
0926 12:34:13.513033 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:13.513036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:13.513088 (+ 52us) server_negotiation.cc:407] Connection header received
0926 12:34:13.513232 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:13.513234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:13.513275 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:13.513356 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:13.513955 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:13.514574 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:13.515466 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:13.515693 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:13.515778 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:13.518570 (+ 2792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.1:55165: BlockingRecv error: recv got EOF from 127.10.98.1:55165 (error 108)
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":74,"threads_started":1}
Sep 26 12:34:13 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:13 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:13.685812 19842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:13.680003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.2:56953)
0926 12:34:13.680331 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:13.680343 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:13.680476 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:13.680771 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:13.680773 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:13.680990 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:13.681169 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:13.681178 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:13.682020 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:13.682023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:13.682654 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:13.682664 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:13.683098 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:13.683101 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:13.683166 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:13.683669 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:13.683680 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:13.685694 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":99,"threads_started":1}
W20260926 12:34:13.686045 19841 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:13.680130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:56953 (local address 127.10.98.62:34931)
0926 12:34:13.680271 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:13.680275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:13.680407 (+ 132us) server_negotiation.cc:407] Connection header received
0926 12:34:13.680590 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:13.680594 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:13.680651 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:13.680727 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:13.681277 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:13.681891 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:13.682771 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:13.682987 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:13.683072 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:13.685851 (+ 2779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.2:56953: BlockingRecv error: recv got EOF from 127.10.98.2:56953 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 12:34:13 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:13 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:13.872182 19843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:13.866560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.3:42697)
0926 12:34:13.866851 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:13.866867 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:13.866979 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:13.867258 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:13.867261 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:13.867452 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:13.867543 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:13.867549 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:13.868371 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:13.868374 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:13.869188 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:13.869199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:13.869705 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:13.869707 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:13.869774 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:13.870287 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:13.870298 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:13.872070 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":95,"threads_started":1}
W20260926 12:34:13.872397 19841 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:13.866725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:42697 (local address 127.10.98.62:34931)
0926 12:34:13.866852 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:13.866856 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:13.866935 (+ 79us) server_negotiation.cc:407] Connection header received
0926 12:34:13.867095 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:13.867098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:13.867146 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:13.867215 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:13.867704 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:13.868256 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:13.869303 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:13.869602 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:13.869686 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:13.872213 (+ 2527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.3:42697: BlockingRecv error: recv got EOF from 127.10.98.3:42697 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 26 12:34:14 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:14 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:14.524793 19844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:14.519385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.1:39349)
0926 12:34:14.519667 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:14.519680 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:14.519775 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:14.520040 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:14.520043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:14.520263 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:14.520367 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:14.520374 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:14.521257 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:14.521260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:14.521934 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:14.521946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:14.522403 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:14.522406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:14.522466 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:14.522915 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:14.522925 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:14.524679 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":82,"threads_started":1}
W20260926 12:34:14.524996 19845 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:14.519476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:39349 (local address 127.10.98.62:34931)
0926 12:34:14.519767 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:14.519770 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:14.519786 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:34:14.519870 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:14.519873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:14.519916 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:14.519985 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:14.520507 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:14.521155 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:14.522069 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:14.522306 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:14.522398 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:14.524809 (+ 2411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.1:39349: BlockingRecv error: recv got EOF from 127.10.98.1:39349 (error 108)
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":76,"threads_started":1}
Sep 26 12:34:14 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:14 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:14.692685 19846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:14.686730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.2:42573)
0926 12:34:14.687005 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:14.687021 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:14.687147 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:14.687407 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:14.687410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:14.687716 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:14.687830 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:14.687837 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:14.688620 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:14.688623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:14.689267 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:14.689276 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:14.689704 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:14.689708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:14.689791 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:14.690324 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:14.690337 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:14.692534 (+ 2197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":89,"threads_started":1}
W20260926 12:34:14.692999 19845 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:14.686829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:42573 (local address 127.10.98.62:34931)
0926 12:34:14.686974 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:14.686979 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:14.687129 (+ 150us) server_negotiation.cc:407] Connection header received
0926 12:34:14.687253 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:14.687256 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:14.687305 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:14.687365 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:14.687960 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:14.688503 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:14.689375 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:14.689602 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:14.689692 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:14.692721 (+ 3029us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.2:42573: BlockingRecv error: recv got EOF from 127.10.98.2:42573 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 26 12:34:14 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:14 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:14.878445 19847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:14.872921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.3:33509)
0926 12:34:14.873290 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:14.873302 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:14.873415 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:14.873713 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:14.873715 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:14.873886 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:14.873980 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:14.873985 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:14.874688 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:14.874690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:14.875387 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:14.875399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:14.875910 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:14.875914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:14.875997 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:14.876506 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:14.876519 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:14.878310 (+ 1791us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":108,"threads_started":1}
W20260926 12:34:14.878544 19845 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:14.873090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:33509 (local address 127.10.98.62:34931)
0926 12:34:14.873271 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:14.873274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:14.873386 (+ 112us) server_negotiation.cc:407] Connection header received
0926 12:34:14.873536 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:14.873539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:14.873596 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:14.873672 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:14.874081 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:14.874583 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:14.875485 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:14.875791 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:14.875893 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:14.878446 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.3:33509: BlockingRecv error: recv got EOF from 127.10.98.3:33509 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 12:34:15 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:15 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:15.531432 19848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:15.525539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.1:35867)
0926 12:34:15.525858 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:15.525871 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:15.525997 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:15.526372 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:15.526375 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:15.526559 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:15.526662 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:15.526669 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:15.527779 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:15.527781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:15.528343 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:15.528351 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:15.528945 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:15.528948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:15.529017 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:15.529612 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:15.529624 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:15.531320 (+ 1696us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":95,"threads_started":1}
W20260926 12:34:15.531646 19849 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:15.525620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:35867 (local address 127.10.98.62:34931)
0926 12:34:15.525872 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:15.525877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:15.525919 (+ 42us) server_negotiation.cc:407] Connection header received
0926 12:34:15.526096 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:15.526099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:15.526154 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:15.526315 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:15.526774 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:15.527650 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:15.528445 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:15.528805 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:15.528908 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:15.531495 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.1:35867: BlockingRecv error: recv got EOF from 127.10.98.1:35867 (error 108)
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":63,"threads_started":1}
Sep 26 12:34:15 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:15 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:15.704061 19850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:15.693509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.2:47869)
0926 12:34:15.693873 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:15.693887 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:15.694014 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:15.694263 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:15.694265 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:15.694563 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:15.694688 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:15.694695 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:15.695480 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:15.695484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:15.696063 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:15.696073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:15.696534 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:15.696538 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:15.696629 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:15.702013 (+ 5384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:15.702027 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:15.703896 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":147,"threads_started":1}
W20260926 12:34:15.704177 19849 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:15.693623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:47869 (local address 127.10.98.62:34931)
0926 12:34:15.693811 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:15.693817 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:15.693971 (+ 154us) server_negotiation.cc:407] Connection header received
0926 12:34:15.694106 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:15.694109 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:15.694155 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:15.694225 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:15.694839 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:15.695369 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:15.696184 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:15.696409 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:15.696500 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:15.704060 (+ 7560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.2:47869: BlockingRecv error: recv got EOF from 127.10.98.2:47869 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 12:34:15 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:15 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:15.885249 19851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:15.879303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.3:43077)
0926 12:34:15.879570 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:15.879584 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:15.879705 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:15.879928 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:15.879930 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:15.880140 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:15.880264 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:15.880272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:15.881243 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:15.881246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:15.881826 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:15.881835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:15.882334 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:15.882337 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:15.882423 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:15.882959 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:15.882976 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:15.885071 (+ 2095us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":74,"threads_started":1}
W20260926 12:34:15.885344 19849 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:15.879386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:43077 (local address 127.10.98.62:34931)
0926 12:34:15.879523 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:15.879528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:15.879658 (+ 130us) server_negotiation.cc:407] Connection header received
0926 12:34:15.879782 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:15.879785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:15.879830 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:15.879899 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:15.880404 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:15.881079 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:15.881930 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:15.882203 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:15.882305 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:15.885241 (+ 2936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.3:43077: BlockingRecv error: recv got EOF from 127.10.98.3:43077 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 12:34:16 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:16 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:16.544982 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:16.532468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.1:40211)
0926 12:34:16.532738 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:16.532752 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:16.532845 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:16.533176 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:16.533179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:16.533387 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:16.533529 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:16.533539 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:16.534562 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:16.534565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:16.535135 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:16.535143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:16.535671 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:16.535674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:16.535749 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:16.542799 (+ 7050us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:16.542814 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:16.544809 (+ 1995us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":77,"threads_started":1}
W20260926 12:34:16.545257 19853 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:16.532638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:40211 (local address 127.10.98.62:34931)
0926 12:34:16.532897 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:16.532901 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:16.532917 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:34:16.532973 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:16.532975 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:16.533018 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:16.533131 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:16.533701 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:16.534450 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:16.535268 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:16.535510 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:16.535602 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:16.545059 (+ 9457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.1:40211: BlockingRecv error: recv got EOF from 127.10.98.1:40211 (error 108)
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":71,"threads_started":1}
Sep 26 12:34:16 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:16 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:16.710546 19854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:16.705010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.2:39237)
0926 12:34:16.705327 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:16.705341 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:16.705434 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:16.705737 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:16.705740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:16.705922 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:16.706022 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:16.706028 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:16.706850 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:16.706853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:16.707435 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:16.707445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:16.707956 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:16.707959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:16.708030 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:16.708539 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:16.708551 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:16.710414 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":130,"threads_started":1}
W20260926 12:34:16.710682 19853 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:16.705224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:39237 (local address 127.10.98.62:34931)
0926 12:34:16.705361 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:16.705365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:16.705389 (+ 24us) server_negotiation.cc:407] Connection header received
0926 12:34:16.705535 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:16.705538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:16.705588 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:16.705669 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:16.706172 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:16.706718 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:16.707600 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:16.707826 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:16.707913 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:16.710561 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.2:39237: BlockingRecv error: recv got EOF from 127.10.98.2:39237 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 12:34:16 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:16 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:16.891889 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:16.886146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.3:49187)
0926 12:34:16.886424 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:16.886438 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:16.886556 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:16.886841 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:16.886845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:16.887049 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:16.887230 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:16.887239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:16.887997 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:16.888001 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:16.888563 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:16.888571 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:16.889009 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:16.889013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:16.889093 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:16.889759 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:16.889775 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:16.891707 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":90,"threads_started":1}
W20260926 12:34:16.891955 19853 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:16.886245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:49187 (local address 127.10.98.62:34931)
0926 12:34:16.886396 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:16.886400 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:16.886508 (+ 108us) server_negotiation.cc:407] Connection header received
0926 12:34:16.886682 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:16.886685 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:16.886732 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:16.886802 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:16.887358 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:16.887889 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:16.888678 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:16.888902 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:16.888993 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:16.891854 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.3:49187: BlockingRecv error: recv got EOF from 127.10.98.3:49187 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 12:34:17 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:17 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:17.551574 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:17.546028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.1:56609)
0926 12:34:17.546434 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:17.546449 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:17.546547 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:17.546823 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:17.546826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:17.547030 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:17.547126 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:17.547132 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:17.547895 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:17.547899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:17.548468 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:17.548476 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:17.548928 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:17.548931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:17.549024 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:17.549543 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:17.549556 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:17.551405 (+ 1849us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":84,"threads_started":1}
W20260926 12:34:17.551702 19857 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:17.546102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:56609 (local address 127.10.98.62:34931)
0926 12:34:17.546381 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:17.546384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:17.546517 (+ 133us) server_negotiation.cc:407] Connection header received
0926 12:34:17.546642 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:17.546645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:17.546695 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:17.546774 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:17.547244 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:17.547785 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:17.548580 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:17.548809 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:17.548897 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:17.551566 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.1:56609: BlockingRecv error: recv got EOF from 127.10.98.1:56609 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":75,"threads_started":1}
Sep 26 12:34:17 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:17 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:17.716962 19858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:17.711530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.2:59981)
0926 12:34:17.711811 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:17.711825 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:17.711921 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:17.712172 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:17.712174 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:17.712352 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:17.712448 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:17.712453 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:17.713258 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:17.713261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:17.714024 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:17.714034 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:17.714504 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:17.714507 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:17.714566 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:17.715026 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:17.715036 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:17.716834 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":83,"threads_started":1}
W20260926 12:34:17.717083 19857 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:17.711707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:59981 (local address 127.10.98.62:34931)
0926 12:34:17.711844 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:17.711847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:17.711879 (+ 32us) server_negotiation.cc:407] Connection header received
0926 12:34:17.712002 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:17.712004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:17.712058 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:17.712139 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:17.712570 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:17.713126 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:17.714162 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:17.714395 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:17.714476 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:17.716968 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.2:59981: BlockingRecv error: recv got EOF from 127.10.98.2:59981 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 12:34:17 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:17 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.3@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:17.898197 19859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:17.892744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.3:36011)
0926 12:34:17.893049 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:17.893064 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:17.893190 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:17.893510 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:17.893512 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:17.893695 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:17.893802 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:17.893808 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:17.894580 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:17.894583 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:17.895184 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:17.895193 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:17.895586 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:17.895588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:17.895679 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:17.896143 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:17.896155 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:17.898025 (+ 1870us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":130,"threads_started":1}
W20260926 12:34:17.898276 19857 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:17.892819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:36011 (local address 127.10.98.62:34931)
0926 12:34:17.892967 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:17.892971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:17.893145 (+ 174us) server_negotiation.cc:407] Connection header received
0926 12:34:17.893264 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:17.893266 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:17.893314 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:17.893387 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:17.893921 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:17.894465 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:17.895284 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:17.895500 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:17.895577 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:17.898175 (+ 2598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.3:36011: BlockingRecv error: recv got EOF from 127.10.98.3:36011 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 12:34:18 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:18 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.1@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:18.558813 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:18.552812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.1:48109)
0926 12:34:18.553172 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:18.553189 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:18.553286 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:18.553658 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:18.553660 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:18.553853 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:18.553950 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:18.553955 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:18.554921 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:18.554925 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:18.555649 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:18.555662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:18.556130 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:18.556134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:18.556245 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:18.556864 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:18.556876 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:18.558681 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":113,"threads_started":1}
W20260926 12:34:18.559026 19861 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:18.552993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:48109 (local address 127.10.98.62:34931)
0926 12:34:18.553249 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:18.553252 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:18.553270 (+ 18us) server_negotiation.cc:407] Connection header received
0926 12:34:18.553384 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:18.553387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:18.553519 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:18.553609 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:18.554073 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:18.554773 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:18.555747 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:18.555981 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:18.556079 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:18.558841 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.1:48109: BlockingRecv error: recv got EOF from 127.10.98.1:48109 (error 108)
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":67,"threads_started":1}
Sep 26 12:34:18 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 12:34:18 dist-test-slave-t7tk krb5kdc[18803](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.10.98.2@KRBTEST.COM for oryx/127.10.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 12:34:18.723505 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:18.718126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34931 (local address 127.10.98.2:36463)
0926 12:34:18.718386 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:18.718400 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:18.718514 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:18.718802 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:18.718805 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:18.718987 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:18.719085 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:18.719091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:18.719876 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:18.719879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:18.720463 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:18.720472 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:18.720933 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:18.720937 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:18.721006 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:18.721605 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:18.721617 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:18.723374 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.10.98.62:34931: Server oryx/127.10.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":77,"threads_started":1}
W20260926 12:34:18.723773 19861 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:18.718231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:36463 (local address 127.10.98.62:34931)
0926 12:34:18.718377 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:18.718381 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:18.718481 (+ 100us) server_negotiation.cc:407] Connection header received
0926 12:34:18.718595 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:18.718597 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:18.718644 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:18.718721 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:18.719199 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:18.719749 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:18.720573 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:18.720810 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:18.720889 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:18.723645 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.10.98.2:36463: BlockingRecv error: recv got EOF from 127.10.98.2:36463 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 12:34:18.792696 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 19386
I20260926 12:34:18.797884 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 19538
I20260926 12:34:18.802650 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 19670
I20260926 12:34:18.807358 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 19316
2026-09-26T12:34:18Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17869 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T12:34:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:34:18Z Disabled control of system clock
I20260926 12:34:18.835959 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:43625
--webserver_interface=127.10.98.62
--webserver_port=0
--builtin_ntp_servers=127.10.98.20:43307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:43625
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 12:34:18.949350 19866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:34:18.949663 19866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:34:18.949708 19866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:34:18.953984 19866 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:34:18.954061 19866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:34:18.954074 19866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:34:18.954093 19866 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:34:18.954107 19866 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:34:18.959084 19866 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:43307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:43625
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:43625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19866
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:34:18.960207 19866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:34:18.961714 19866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:34:18.971171 19874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:18.971241 19871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:18.971293 19866 server_base.cc:1034] running on GCE node
W20260926 12:34:18.971300 19872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:18.972085 19866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:34:18.973394 19866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:34:18.974578 19866 hybrid_clock.cc:648] HybridClock initialized: now 1790426058974557 us; error 36 us; skew 500 ppm
I20260926 12:34:18.976480 19866 webserver.cc:460] Webserver started at http://127.10.98.62:35797/ using document root <none> and password file <none>
I20260926 12:34:18.977051 19866 fs_manager.cc:359] Metadata directory not provided
I20260926 12:34:18.977164 19866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:34:18.977373 19866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:34:18.979415 19866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "f14db799281842ba8a4bcfeee6c21f33"
format_stamp: "Formatted at 2026-09-26 12:34:18 on dist-test-slave-t7tk"
server_key: "8451556ea987719beea8200b2a26d54f"
server_key_iv: "0f394139681786ee1347de28aa442998"
server_key_version: "encryptionkey@0"
I20260926 12:34:18.979969 19866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "f14db799281842ba8a4bcfeee6c21f33"
format_stamp: "Formatted at 2026-09-26 12:34:18 on dist-test-slave-t7tk"
server_key: "8451556ea987719beea8200b2a26d54f"
server_key_iv: "0f394139681786ee1347de28aa442998"
server_key_version: "encryptionkey@0"
I20260926 12:34:18.983958 19866 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260926 12:34:18.986649 19880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:18.987946 19866 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:34:18.988070 19866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "f14db799281842ba8a4bcfeee6c21f33"
format_stamp: "Formatted at 2026-09-26 12:34:18 on dist-test-slave-t7tk"
server_key: "8451556ea987719beea8200b2a26d54f"
server_key_iv: "0f394139681786ee1347de28aa442998"
server_key_version: "encryptionkey@0"
I20260926 12:34:18.988185 19866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:34:19.004137 19866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:34:19.004875 19866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:34:19.005051 19866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:34:19.011685 19866 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:43625
I20260926 12:34:19.011708 19932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:43625 every 8 connection(s)
I20260926 12:34:19.012598 19866 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:34:19.015951 19933 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:34:19.021852 19933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33: Bootstrap starting.
I20260926 12:34:19.021991 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 19866
I20260926 12:34:19.022095 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:34:19.022368 10632 external_mini_cluster.cc:1428] Setting key ae7b7f4483ad5bb1c4820a21000cff65
I20260926 12:34:19.024614 19933 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33: Neither blocks nor log segments found. Creating new log.
I20260926 12:34:19.025596 19933 log.cc:824] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33: Log is configured to *not* fsync() on all Append() calls
I20260926 12:34:19.027966 19933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33: No bootstrap required, opened a new log
I20260926 12:34:19.032449 19933 raft_consensus.cc:348] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f14db799281842ba8a4bcfeee6c21f33" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43625 } }
I20260926 12:34:19.032780 19933 raft_consensus.cc:374] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:34:19.032840 19933 raft_consensus.cc:729] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f14db799281842ba8a4bcfeee6c21f33, State: Initialized, Role: FOLLOWER
I20260926 12:34:19.033370 19933 consensus_queue.cc:260] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f14db799281842ba8a4bcfeee6c21f33" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43625 } }
I20260926 12:34:19.033550 19933 raft_consensus.cc:388] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:34:19.033627 19933 raft_consensus.cc:482] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:34:19.033717 19933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:34:19.033747 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:19.023884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41308 (local address 127.10.98.62:43625)
0926 12:34:19.024654 (+ 770us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:19.024679 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:19.024726 (+ 47us) server_negotiation.cc:407] Connection header received
0926 12:34:19.025632 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:19.025659 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:19.026182 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:19.026540 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:19.026999 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:19.028538 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:19.029552 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:19.030312 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:19.030477 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:19.031424 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:34:19.031431 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:34:19.031451 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:34:19.032007 (+ 556us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:34:19.032082 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:34:19.032108 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:34:19.032206 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:19.032360 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 12:34:19.032653 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":132,"threads_started":1}
I20260926 12:34:19.034816 19933 raft_consensus.cc:504] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f14db799281842ba8a4bcfeee6c21f33" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43625 } }
I20260926 12:34:19.035200 19933 leader_election.cc:302] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f14db799281842ba8a4bcfeee6c21f33; no voters:
I20260926 12:34:19.035568 19933 leader_election.cc:288] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:34:19.035769 19938 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:34:19.035974 19938 raft_consensus.cc:686] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33 [term 1 LEADER]: Becoming Leader. State: Replica: f14db799281842ba8a4bcfeee6c21f33, State: Running, Role: LEADER
I20260926 12:34:19.036439 19938 consensus_queue.cc:237] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f14db799281842ba8a4bcfeee6c21f33" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43625 } }
I20260926 12:34:19.036983 19933 sys_catalog.cc:564] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:34:19.037709 19940 sys_catalog.cc:455] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f14db799281842ba8a4bcfeee6c21f33. Latest consensus state: current_term: 1 leader_uuid: "f14db799281842ba8a4bcfeee6c21f33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f14db799281842ba8a4bcfeee6c21f33" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43625 } } }
I20260926 12:34:19.037832 19940 sys_catalog.cc:458] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33 [sys.catalog]: This master's current role is: LEADER
I20260926 12:34:19.038079 19939 sys_catalog.cc:455] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f14db799281842ba8a4bcfeee6c21f33" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f14db799281842ba8a4bcfeee6c21f33" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43625 } } }
I20260926 12:34:19.038268 19939 sys_catalog.cc:458] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33 [sys.catalog]: This master's current role is: LEADER
I20260926 12:34:19.038337 19946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:34:19.041687 19946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:34:19.047536 19946 catalog_manager.cc:1347] Generated new cluster ID: 69ec0b09d7584e9589bc7c2c4f961b96
I20260926 12:34:19.047621 19946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:34:19.065158 19946 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:34:19.066170 19946 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:34:19.083714 19946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f14db799281842ba8a4bcfeee6c21f33: Generated new TSK 0
I20260926 12:34:19.084482 19946 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:34:19.092710 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:43625
--builtin_ntp_servers=127.10.98.20:43307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 12:34:19.207387 19957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:34:19.207708 19957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:34:19.207747 19957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:34:19.211933 19957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:34:19.212010 19957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:34:19.212064 19957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:34:19.217145 19957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:43307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:43625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:34:19.218307 19957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:34:19.219764 19957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:34:19.229727 19963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:19.229730 19962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:19.229730 19965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:19.230057 19957 server_base.cc:1034] running on GCE node
I20260926 12:34:19.230501 19957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:34:19.231145 19957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:34:19.232331 19957 hybrid_clock.cc:648] HybridClock initialized: now 1790426059232309 us; error 34 us; skew 500 ppm
I20260926 12:34:19.234318 19957 webserver.cc:460] Webserver started at http://127.10.98.1:46793/ using document root <none> and password file <none>
I20260926 12:34:19.234890 19957 fs_manager.cc:359] Metadata directory not provided
I20260926 12:34:19.234952 19957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:34:19.235164 19957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:34:19.237317 19957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "28bda36e0eee4772ace5f08660f2256b"
format_stamp: "Formatted at 2026-09-26 12:34:19 on dist-test-slave-t7tk"
server_key: "73342a62bb486299f40e9132f401ab28"
server_key_iv: "8ea1b41bdf9b2bde9f3d9a5ae09c16a4"
server_key_version: "encryptionkey@0"
I20260926 12:34:19.237939 19957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "28bda36e0eee4772ace5f08660f2256b"
format_stamp: "Formatted at 2026-09-26 12:34:19 on dist-test-slave-t7tk"
server_key: "73342a62bb486299f40e9132f401ab28"
server_key_iv: "8ea1b41bdf9b2bde9f3d9a5ae09c16a4"
server_key_version: "encryptionkey@0"
I20260926 12:34:19.241906 19957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 12:34:19.244629 19971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:19.246071 19957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:34:19.246217 19957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "28bda36e0eee4772ace5f08660f2256b"
format_stamp: "Formatted at 2026-09-26 12:34:19 on dist-test-slave-t7tk"
server_key: "73342a62bb486299f40e9132f401ab28"
server_key_iv: "8ea1b41bdf9b2bde9f3d9a5ae09c16a4"
server_key_version: "encryptionkey@0"
I20260926 12:34:19.246330 19957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:34:19.259086 19957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:34:19.259801 19957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:34:19.259979 19957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:34:19.260564 19957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:34:19.261566 19957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:34:19.261615 19957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:19.261668 19957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:34:19.261698 19957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:19.270080 19957 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:42607
I20260926 12:34:19.270136 20084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:42607 every 8 connection(s)
I20260926 12:34:19.270989 19957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:34:19.279496 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 19957
I20260926 12:34:19.279593 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:34:19.279857 10632 external_mini_cluster.cc:1428] Setting key 591e0048916248b3de24bb18de2b8102
I20260926 12:34:19.280397 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:19.272762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:49891 (local address 127.10.98.62:43625)
0926 12:34:19.272887 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:19.272891 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:19.274093 (+ 1202us) server_negotiation.cc:407] Connection header received
0926 12:34:19.275517 (+ 1424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:19.275520 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:19.275581 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:19.275664 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:19.276746 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:19.277340 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:19.278486 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:19.278720 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:19.278809 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:19.279898 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:34:19.279902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:34:19.279905 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:34:19.279921 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:34:19.279944 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:34:19.279954 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:34:19.280015 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:19.280188 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 12:34:19.280254 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:34:19.281502 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:19.273227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:43625 (local address 127.10.98.1:49891)
0926 12:34:19.273939 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:19.273982 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:19.275289 (+ 1307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:19.275855 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:19.275863 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:19.275955 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:19.276529 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:19.276543 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:19.277464 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:19.277466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:19.278356 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:19.278366 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:19.278870 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:19.278874 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:19.279031 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:19.279625 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:34:19.279628 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:19.279677 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:34:19.279745 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:34:19.280038 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:34:19.280047 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:34:19.280054 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 12:34:19.280172 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 12:34:19.280469 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":110,"threads_started":1}
I20260926 12:34:19.282059 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:43625
--builtin_ntp_servers=127.10.98.20:43307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 12:34:19.282647 20085 heartbeater.cc:347] Connected to a master server at 127.10.98.62:43625
I20260926 12:34:19.282917 20085 heartbeater.cc:464] Registering TS with master...
I20260926 12:34:19.283548 20085 heartbeater.cc:511] Master 127.10.98.62:43625 requested a full tablet report, sending...
I20260926 12:34:19.285039 19897 ts_manager.cc:194] Registered new tserver with Master: 28bda36e0eee4772ace5f08660f2256b (127.10.98.1:42607)
I20260926 12:34:19.286209 19897 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.98.1:49891
W20260926 12:34:19.399214 20088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:34:19.399519 20088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:34:19.399561 20088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:34:19.403788 20088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:34:19.403851 20088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:34:19.403906 20088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:34:19.408778 20088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:43307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:43625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:34:19.409900 20088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:34:19.411248 20088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:34:19.421186 20094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:19.421253 20088 server_base.cc:1034] running on GCE node
W20260926 12:34:19.421200 20093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:19.421228 20096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:19.421837 20088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:34:19.422516 20088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:34:19.423702 20088 hybrid_clock.cc:648] HybridClock initialized: now 1790426059423668 us; error 47 us; skew 500 ppm
I20260926 12:34:19.425717 20088 webserver.cc:460] Webserver started at http://127.10.98.2:46115/ using document root <none> and password file <none>
I20260926 12:34:19.426307 20088 fs_manager.cc:359] Metadata directory not provided
I20260926 12:34:19.426379 20088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:34:19.426594 20088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:34:19.428735 20088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "2579858bbc134288846f1b919ec5f876"
format_stamp: "Formatted at 2026-09-26 12:34:19 on dist-test-slave-t7tk"
server_key: "739465d27f7cf19edd1933e16269c402"
server_key_iv: "639add70be25ef92ff5a132796fc0d7c"
server_key_version: "encryptionkey@0"
I20260926 12:34:19.429345 20088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "2579858bbc134288846f1b919ec5f876"
format_stamp: "Formatted at 2026-09-26 12:34:19 on dist-test-slave-t7tk"
server_key: "739465d27f7cf19edd1933e16269c402"
server_key_iv: "639add70be25ef92ff5a132796fc0d7c"
server_key_version: "encryptionkey@0"
I20260926 12:34:19.433331 20088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:34:19.436092 20102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:19.437621 20088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:34:19.437774 20088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "2579858bbc134288846f1b919ec5f876"
format_stamp: "Formatted at 2026-09-26 12:34:19 on dist-test-slave-t7tk"
server_key: "739465d27f7cf19edd1933e16269c402"
server_key_iv: "639add70be25ef92ff5a132796fc0d7c"
server_key_version: "encryptionkey@0"
I20260926 12:34:19.437886 20088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:34:19.446712 20088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:34:19.447361 20088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:34:19.447536 20088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:34:19.448086 20088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:34:19.449024 20088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:34:19.449070 20088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:19.449138 20088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:34:19.449165 20088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:19.457938 20088 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:44357
I20260926 12:34:19.457965 20215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:44357 every 8 connection(s)
I20260926 12:34:19.458864 20088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:34:19.468729 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:19.460852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:35263 (local address 127.10.98.62:43625)
0926 12:34:19.461044 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:19.461048 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:19.462331 (+ 1283us) server_negotiation.cc:407] Connection header received
0926 12:34:19.463732 (+ 1401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:19.463736 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:19.463802 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:19.463897 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:19.464892 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:19.465514 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:19.466740 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:19.466976 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:19.467067 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:19.468253 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:34:19.468257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:34:19.468259 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:34:19.468273 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:34:19.468292 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:34:19.468301 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:34:19.468358 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:19.468504 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 12:34:19.468572 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:34:19.468852 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 20088
I20260926 12:34:19.468954 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:34:19.469327 10632 external_mini_cluster.cc:1428] Setting key 59be4ff85556dbb4f73319cb4843ee28
I20260926 12:34:19.469978 20218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:19.461403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:43625 (local address 127.10.98.2:35263)
0926 12:34:19.462172 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:19.462218 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:19.463479 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:19.464063 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:19.464069 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:19.464161 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:19.464723 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:19.464738 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:19.465699 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:19.465702 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:19.466602 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:19.466611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:19.467156 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:19.467159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:19.467332 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:19.467970 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:34:19.467972 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:19.468015 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:34:19.468087 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:34:19.468368 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:34:19.468374 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:34:19.468378 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:34:19.468475 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 12:34:19.468810 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":124,"threads_started":1}
I20260926 12:34:19.471283 20216 heartbeater.cc:347] Connected to a master server at 127.10.98.62:43625
I20260926 12:34:19.471517 20216 heartbeater.cc:464] Registering TS with master...
I20260926 12:34:19.471652 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:43625
--builtin_ntp_servers=127.10.98.20:43307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 12:34:19.472081 20216 heartbeater.cc:511] Master 127.10.98.62:43625 requested a full tablet report, sending...
I20260926 12:34:19.473223 19896 ts_manager.cc:194] Registered new tserver with Master: 2579858bbc134288846f1b919ec5f876 (127.10.98.2:44357)
I20260926 12:34:19.473820 19896 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.98.2:35263
W20260926 12:34:19.586400 20219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:34:19.586679 20219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:34:19.586716 20219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:34:19.590893 20219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:34:19.590965 20219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:34:19.591022 20219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:34:19.596015 20219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:43307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:43625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:34:19.597184 20219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:34:19.598628 20219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:34:19.608238 20224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:19.608340 20219 server_base.cc:1034] running on GCE node
W20260926 12:34:19.608270 20227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:19.608283 20225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:19.608889 20219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:34:19.609673 20219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:34:19.610891 20219 hybrid_clock.cc:648] HybridClock initialized: now 1790426059610869 us; error 43 us; skew 500 ppm
I20260926 12:34:19.612942 20219 webserver.cc:460] Webserver started at http://127.10.98.3:44735/ using document root <none> and password file <none>
I20260926 12:34:19.613583 20219 fs_manager.cc:359] Metadata directory not provided
I20260926 12:34:19.613656 20219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:34:19.613858 20219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:34:19.615928 20219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "ca6698f1855b480391116b1b75162101"
format_stamp: "Formatted at 2026-09-26 12:34:19 on dist-test-slave-t7tk"
server_key: "5232fcab505a3406552e3d501dc068b1"
server_key_iv: "c03c0a5593d7ff677e9049cb75fdcdb2"
server_key_version: "encryptionkey@0"
I20260926 12:34:19.616480 20219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "ca6698f1855b480391116b1b75162101"
format_stamp: "Formatted at 2026-09-26 12:34:19 on dist-test-slave-t7tk"
server_key: "5232fcab505a3406552e3d501dc068b1"
server_key_iv: "c03c0a5593d7ff677e9049cb75fdcdb2"
server_key_version: "encryptionkey@0"
I20260926 12:34:19.620286 20219 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:34:19.623268 20233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:19.624661 20219 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:34:19.624848 20219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "ca6698f1855b480391116b1b75162101"
format_stamp: "Formatted at 2026-09-26 12:34:19 on dist-test-slave-t7tk"
server_key: "5232fcab505a3406552e3d501dc068b1"
server_key_iv: "c03c0a5593d7ff677e9049cb75fdcdb2"
server_key_version: "encryptionkey@0"
I20260926 12:34:19.624962 20219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:34:19.638393 20219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:34:19.639122 20219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:34:19.639307 20219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:34:19.639896 20219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:34:19.640880 20219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:34:19.640928 20219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:19.640980 20219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:34:19.641008 20219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:19.649431 20219 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:41721
I20260926 12:34:19.649451 20346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:41721 every 8 connection(s)
I20260926 12:34:19.650534 20219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:34:19.658368 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 20219
I20260926 12:34:19.658470 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:34:19.658748 10632 external_mini_cluster.cc:1428] Setting key 7818d6817a701e2c7f04177a37ea429b
I20260926 12:34:19.660075 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:19.652523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:36445 (local address 127.10.98.62:43625)
0926 12:34:19.652647 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:19.652651 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:19.653807 (+ 1156us) server_negotiation.cc:407] Connection header received
0926 12:34:19.655284 (+ 1477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:19.655287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:19.655351 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:19.655432 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:19.656388 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:19.656923 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:19.658121 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:19.658349 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:19.658439 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:19.659601 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:34:19.659605 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:34:19.659608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:34:19.659621 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:34:19.659644 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:34:19.659655 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:34:19.659731 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:19.659875 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 12:34:19.659938 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:34:19.660985 20349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:19.652897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:43625 (local address 127.10.98.3:36445)
0926 12:34:19.653619 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:19.653662 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:19.655041 (+ 1379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:19.655633 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:19.655641 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:19.655743 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:19.656245 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:19.656258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:19.657048 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:19.657051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:19.657999 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:19.658009 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:19.658502 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:19.658506 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:19.658660 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:19.659313 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:34:19.659314 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:19.659358 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:34:19.659441 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:34:19.659734 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:34:19.659744 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:34:19.659750 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 12:34:19.659868 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 12:34:19.660168 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":141,"threads_started":1}
I20260926 12:34:19.662169 20347 heartbeater.cc:347] Connected to a master server at 127.10.98.62:43625
I20260926 12:34:19.662396 20347 heartbeater.cc:464] Registering TS with master...
I20260926 12:34:19.662967 20347 heartbeater.cc:511] Master 127.10.98.62:43625 requested a full tablet report, sending...
I20260926 12:34:19.664000 19896 ts_manager.cc:194] Registered new tserver with Master: ca6698f1855b480391116b1b75162101 (127.10.98.3:41721)
I20260926 12:34:19.664608 19896 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.98.3:36445
I20260926 12:34:19.672813 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 12:34:19.675963 19936 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:19.674967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41310 (local address 127.10.98.62:43625)
0926 12:34:19.675154 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:19.675160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:19.675298 (+ 138us) server_negotiation.cc:407] Connection header received
0926 12:34:19.675440 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:19.675443 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:19.675491 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:19.675556 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:19.675826 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41310: BlockingRecv error: recv got EOF from 127.0.0.1:41310 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260926 12:34:19.676434 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 19957
I20260926 12:34:19.681272 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 20088
I20260926 12:34:19.687449 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 20219
I20260926 12:34:19.692461 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 19866
2026-09-26T12:34:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (884 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T12:34:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:34:19Z Disabled control of system clock
I20260926 12:34:19.721652 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:41583
--webserver_interface=127.10.98.62
--webserver_port=0
--builtin_ntp_servers=127.10.98.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:41583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 12:34:19.831528 20358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:34:19.831825 20358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:34:19.831862 20358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:34:19.835942 20358 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:34:19.836014 20358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:34:19.836028 20358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:34:19.836045 20358 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:34:19.836057 20358 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:34:19.840961 20358 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:41583
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:41583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20358
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:34:19.842123 20358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:34:19.843451 20358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:34:19.851989 20366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:19.851997 20364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:19.851997 20363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:19.852387 20358 server_base.cc:1034] running on GCE node
I20260926 12:34:19.853020 20358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:34:19.854255 20358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:34:19.855451 20358 hybrid_clock.cc:648] HybridClock initialized: now 1790426059855431 us; error 41 us; skew 500 ppm
I20260926 12:34:19.857556 20358 webserver.cc:460] Webserver started at http://127.10.98.62:40007/ using document root <none> and password file <none>
I20260926 12:34:19.858142 20358 fs_manager.cc:359] Metadata directory not provided
I20260926 12:34:19.858215 20358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:34:19.858412 20358 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:34:19.860515 20358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "262ca769894a462b8dbcd112a8811724"
format_stamp: "Formatted at 2026-09-26 12:34:19 on dist-test-slave-t7tk"
server_key: "a17943cc0b9ee4ac662727c6b7ef6248"
server_key_iv: "6a0afb46e73a23df62a5a17078e61544"
server_key_version: "encryptionkey@0"
I20260926 12:34:19.861088 20358 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "262ca769894a462b8dbcd112a8811724"
format_stamp: "Formatted at 2026-09-26 12:34:19 on dist-test-slave-t7tk"
server_key: "a17943cc0b9ee4ac662727c6b7ef6248"
server_key_iv: "6a0afb46e73a23df62a5a17078e61544"
server_key_version: "encryptionkey@0"
I20260926 12:34:19.865018 20358 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 12:34:19.867692 20372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:19.869067 20358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:34:19.869241 20358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "262ca769894a462b8dbcd112a8811724"
format_stamp: "Formatted at 2026-09-26 12:34:19 on dist-test-slave-t7tk"
server_key: "a17943cc0b9ee4ac662727c6b7ef6248"
server_key_iv: "6a0afb46e73a23df62a5a17078e61544"
server_key_version: "encryptionkey@0"
I20260926 12:34:19.869361 20358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:34:19.874130 20358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:34:19.874818 20358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:34:19.874982 20358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:34:19.881592 20358 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:41583
I20260926 12:34:19.881630 20424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:41583 every 8 connection(s)
I20260926 12:34:19.882578 20358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:34:19.885905 20425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:34:19.887564 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 20358
I20260926 12:34:19.887660 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:34:19.887921 10632 external_mini_cluster.cc:1428] Setting key 8b5369e621b4ce864c0d0dec9dc54862
I20260926 12:34:19.891999 20425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724: Bootstrap starting.
I20260926 12:34:19.894163 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:19.889544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39464 (local address 127.10.98.62:41583)
0926 12:34:19.890267 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:19.890284 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:19.890323 (+ 39us) server_negotiation.cc:407] Connection header received
0926 12:34:19.891103 (+ 780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:19.891124 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:19.891512 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:19.891869 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:19.892722 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:34:19.892727 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:34:19.892741 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:34:19.892751 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:34:19.892810 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:34:19.892827 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:34:19.892960 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:19.893132 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 12:34:19.893417 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":128,"threads_started":1}
I20260926 12:34:19.894625 20425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724: Neither blocks nor log segments found. Creating new log.
I20260926 12:34:19.895481 20425 log.cc:824] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724: Log is configured to *not* fsync() on all Append() calls
I20260926 12:34:19.897651 20425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724: No bootstrap required, opened a new log
I20260926 12:34:19.901252 20425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262ca769894a462b8dbcd112a8811724" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41583 } }
I20260926 12:34:19.901512 20425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:34:19.901561 20425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 262ca769894a462b8dbcd112a8811724, State: Initialized, Role: FOLLOWER
I20260926 12:34:19.902016 20425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262ca769894a462b8dbcd112a8811724" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41583 } }
I20260926 12:34:19.902140 20425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:34:19.902212 20425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:34:19.902305 20425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:34:19.903259 20425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262ca769894a462b8dbcd112a8811724" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41583 } }
I20260926 12:34:19.903586 20425 leader_election.cc:302] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 262ca769894a462b8dbcd112a8811724; no voters:
I20260926 12:34:19.903891 20425 leader_election.cc:288] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:34:19.904014 20430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:34:19.904287 20430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724 [term 1 LEADER]: Becoming Leader. State: Replica: 262ca769894a462b8dbcd112a8811724, State: Running, Role: LEADER
I20260926 12:34:19.904610 20430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262ca769894a462b8dbcd112a8811724" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41583 } }
I20260926 12:34:19.905088 20425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:34:19.906221 20431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "262ca769894a462b8dbcd112a8811724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262ca769894a462b8dbcd112a8811724" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41583 } } }
I20260926 12:34:19.906328 20431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724 [sys.catalog]: This master's current role is: LEADER
I20260926 12:34:19.906605 20432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 262ca769894a462b8dbcd112a8811724. Latest consensus state: current_term: 1 leader_uuid: "262ca769894a462b8dbcd112a8811724" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "262ca769894a462b8dbcd112a8811724" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41583 } } }
I20260926 12:34:19.906710 20439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:34:19.906737 20432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724 [sys.catalog]: This master's current role is: LEADER
I20260926 12:34:19.910559 20439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:34:19.916404 20439 catalog_manager.cc:1347] Generated new cluster ID: ccbc8a07ab7c4d60a47182b87dd0614d
I20260926 12:34:19.916491 20439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:34:19.941360 20439 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:34:19.941763 20439 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:34:19.947450 20439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 262ca769894a462b8dbcd112a8811724: Generated new TSK 0
I20260926 12:34:19.948359 20439 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:34:19.953845 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:41583
--builtin_ntp_servers=127.10.98.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 12:34:20.072765 20449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:34:20.073042 20449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:34:20.073083 20449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:34:20.077301 20449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:34:20.077370 20449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:34:20.077427 20449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:34:20.082546 20449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:41583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:34:20.083691 20449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:34:20.085093 20449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:34:20.095940 20449 server_base.cc:1034] running on GCE node
W20260926 12:34:20.095835 20457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:20.095835 20455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:20.095830 20454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:20.096630 20449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:34:20.097365 20449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:34:20.098568 20449 hybrid_clock.cc:648] HybridClock initialized: now 1790426060098551 us; error 35 us; skew 500 ppm
I20260926 12:34:20.100545 20449 webserver.cc:460] Webserver started at http://127.10.98.1:42925/ using document root <none> and password file <none>
I20260926 12:34:20.101162 20449 fs_manager.cc:359] Metadata directory not provided
I20260926 12:34:20.101248 20449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:34:20.101455 20449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:34:20.103536 20449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "9f4a22e00cd7403eb07c1b4f45281d78"
format_stamp: "Formatted at 2026-09-26 12:34:20 on dist-test-slave-t7tk"
server_key: "e22c620f285bd6869abdf21265848efd"
server_key_iv: "b6d45cff4a1d744f32b043499465020b"
server_key_version: "encryptionkey@0"
I20260926 12:34:20.104089 20449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "9f4a22e00cd7403eb07c1b4f45281d78"
format_stamp: "Formatted at 2026-09-26 12:34:20 on dist-test-slave-t7tk"
server_key: "e22c620f285bd6869abdf21265848efd"
server_key_iv: "b6d45cff4a1d744f32b043499465020b"
server_key_version: "encryptionkey@0"
I20260926 12:34:20.108137 20449 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:34:20.110808 20463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:20.112072 20449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:34:20.112214 20449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "9f4a22e00cd7403eb07c1b4f45281d78"
format_stamp: "Formatted at 2026-09-26 12:34:20 on dist-test-slave-t7tk"
server_key: "e22c620f285bd6869abdf21265848efd"
server_key_iv: "b6d45cff4a1d744f32b043499465020b"
server_key_version: "encryptionkey@0"
I20260926 12:34:20.112325 20449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:34:20.116953 20449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:34:20.117622 20449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:34:20.117795 20449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:34:20.118345 20449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:34:20.119271 20449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:34:20.119318 20449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:20.119369 20449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:34:20.119397 20449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:20.127794 20449 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:39417
I20260926 12:34:20.127838 20576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:39417 every 8 connection(s)
I20260926 12:34:20.128741 20449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:34:20.130362 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 20449
I20260926 12:34:20.130462 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:34:20.130764 10632 external_mini_cluster.cc:1428] Setting key c80648250271fcacb097d8384faea4d7
I20260926 12:34:20.133004 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:41583
--builtin_ntp_servers=127.10.98.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 12:34:20.136035 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:20.130720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:53219 (local address 127.10.98.62:41583)
0926 12:34:20.130893 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:20.130899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:20.132220 (+ 1321us) server_negotiation.cc:407] Connection header received
0926 12:34:20.134120 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:20.134124 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:20.134163 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:20.134245 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:20.135539 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:34:20.135544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:34:20.135547 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:34:20.135562 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:34:20.135588 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:34:20.135598 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:34:20.135762 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:20.135860 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 12:34:20.135904 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 12:34:20.137085 20579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:20.131210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:41583 (local address 127.10.98.1:53219)
0926 12:34:20.132087 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:20.132131 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:20.133783 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:20.134461 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:20.134471 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:20.134624 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:20.135253 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:34:20.135255 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:20.135299 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:34:20.135374 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:34:20.135715 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:34:20.135724 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:34:20.135729 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 12:34:20.135889 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 12:34:20.136101 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":148,"threads_started":1}
I20260926 12:34:20.138327 20577 heartbeater.cc:347] Connected to a master server at 127.10.98.62:41583
I20260926 12:34:20.138574 20577 heartbeater.cc:464] Registering TS with master...
I20260926 12:34:20.139004 20577 heartbeater.cc:511] Master 127.10.98.62:41583 requested a full tablet report, sending...
I20260926 12:34:20.140537 20389 ts_manager.cc:194] Registered new tserver with Master: 9f4a22e00cd7403eb07c1b4f45281d78 (127.10.98.1:39417)
W20260926 12:34:20.245623 20580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:34:20.245918 20580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:34:20.245959 20580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:34:20.250451 20580 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:34:20.250520 20580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:34:20.250599 20580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:34:20.255596 20580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:41583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20580
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:34:20.256712 20580 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:34:20.258206 20580 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:34:20.267788 20588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:20.267833 20585 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:20.267866 20580 server_base.cc:1034] running on GCE node
W20260926 12:34:20.267788 20586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:20.268411 20580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:34:20.269057 20580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:34:20.270267 20580 hybrid_clock.cc:648] HybridClock initialized: now 1790426060270250 us; error 31 us; skew 500 ppm
I20260926 12:34:20.272207 20580 webserver.cc:460] Webserver started at http://127.10.98.2:36209/ using document root <none> and password file <none>
I20260926 12:34:20.272787 20580 fs_manager.cc:359] Metadata directory not provided
I20260926 12:34:20.272856 20580 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:34:20.273066 20580 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:34:20.275211 20580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "8098e327ef5f4e26ab755e9b1ae826a3"
format_stamp: "Formatted at 2026-09-26 12:34:20 on dist-test-slave-t7tk"
server_key: "f14d6e96afc713362d253d9f0258f909"
server_key_iv: "c200f2d48f5e75bcd0182981bc570c2b"
server_key_version: "encryptionkey@0"
I20260926 12:34:20.275864 20580 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "8098e327ef5f4e26ab755e9b1ae826a3"
format_stamp: "Formatted at 2026-09-26 12:34:20 on dist-test-slave-t7tk"
server_key: "f14d6e96afc713362d253d9f0258f909"
server_key_iv: "c200f2d48f5e75bcd0182981bc570c2b"
server_key_version: "encryptionkey@0"
I20260926 12:34:20.279760 20580 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260926 12:34:20.282603 20594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:20.284060 20580 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:34:20.284180 20580 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "8098e327ef5f4e26ab755e9b1ae826a3"
format_stamp: "Formatted at 2026-09-26 12:34:20 on dist-test-slave-t7tk"
server_key: "f14d6e96afc713362d253d9f0258f909"
server_key_iv: "c200f2d48f5e75bcd0182981bc570c2b"
server_key_version: "encryptionkey@0"
I20260926 12:34:20.284291 20580 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:34:20.288852 20580 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:34:20.289530 20580 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:34:20.289690 20580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:34:20.290262 20580 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:34:20.291354 20580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:34:20.291414 20580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:20.291462 20580 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:34:20.291527 20580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:20.299940 20580 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:34409
I20260926 12:34:20.299965 20707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:34409 every 8 connection(s)
I20260926 12:34:20.300833 20580 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:34:20.307371 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:20.302867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:39245 (local address 127.10.98.62:41583)
0926 12:34:20.303042 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:20.303045 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:20.304173 (+ 1128us) server_negotiation.cc:407] Connection header received
0926 12:34:20.305625 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:20.305630 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:20.305688 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:20.305767 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:20.306917 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:34:20.306921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:34:20.306924 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:34:20.306937 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:34:20.306958 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:34:20.306967 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:34:20.307049 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:20.307175 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 12:34:20.307223 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260926 12:34:20.308257 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:20.303289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:41583 (local address 127.10.98.2:39245)
0926 12:34:20.304003 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:20.304048 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:20.305399 (+ 1351us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:20.305959 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:20.305966 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:20.306083 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:20.306638 (+ 555us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:34:20.306640 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:20.306687 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:34:20.306762 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:34:20.307038 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:34:20.307046 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:34:20.307051 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 12:34:20.307223 (+ 172us) client_negotiation.cc:240] Negotiation successful
0926 12:34:20.307484 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":119,"threads_started":1}
I20260926 12:34:20.309442 20708 heartbeater.cc:347] Connected to a master server at 127.10.98.62:41583
I20260926 12:34:20.309724 20708 heartbeater.cc:464] Registering TS with master...
I20260926 12:34:20.309990 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 20580
I20260926 12:34:20.310112 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:34:20.310235 20708 heartbeater.cc:511] Master 127.10.98.62:41583 requested a full tablet report, sending...
I20260926 12:34:20.310443 10632 external_mini_cluster.cc:1428] Setting key db6744bc85ed391c070f17b52872d323
I20260926 12:34:20.311529 20388 ts_manager.cc:194] Registered new tserver with Master: 8098e327ef5f4e26ab755e9b1ae826a3 (127.10.98.2:34409)
I20260926 12:34:20.312656 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:41583
--builtin_ntp_servers=127.10.98.20:39579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 12:34:20.425380 20711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:34:20.425688 20711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:34:20.425746 20711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:34:20.430276 20711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:34:20.430361 20711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:34:20.430445 20711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:34:20.435478 20711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:39579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:41583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:34:20.436642 20711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:34:20.438180 20711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:34:20.449545 20717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:20.449550 20719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:20.449681 20711 server_base.cc:1034] running on GCE node
W20260926 12:34:20.449661 20716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:20.450350 20711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:34:20.451045 20711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:34:20.452231 20711 hybrid_clock.cc:648] HybridClock initialized: now 1790426060452200 us; error 49 us; skew 500 ppm
I20260926 12:34:20.454526 20711 webserver.cc:460] Webserver started at http://127.10.98.3:43331/ using document root <none> and password file <none>
I20260926 12:34:20.455269 20711 fs_manager.cc:359] Metadata directory not provided
I20260926 12:34:20.455351 20711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:34:20.455574 20711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:34:20.457918 20711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "489e925e648e43c894debfcb185270d5"
format_stamp: "Formatted at 2026-09-26 12:34:20 on dist-test-slave-t7tk"
server_key: "a19d0f50a6512e2f1263a53e4ce4e7c4"
server_key_iv: "e5c9ea27b46ff8e533ffc95b61cfaef9"
server_key_version: "encryptionkey@0"
I20260926 12:34:20.458552 20711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "489e925e648e43c894debfcb185270d5"
format_stamp: "Formatted at 2026-09-26 12:34:20 on dist-test-slave-t7tk"
server_key: "a19d0f50a6512e2f1263a53e4ce4e7c4"
server_key_iv: "e5c9ea27b46ff8e533ffc95b61cfaef9"
server_key_version: "encryptionkey@0"
I20260926 12:34:20.462854 20711 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:34:20.465998 20725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:20.467533 20711 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:34:20.467760 20711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "489e925e648e43c894debfcb185270d5"
format_stamp: "Formatted at 2026-09-26 12:34:20 on dist-test-slave-t7tk"
server_key: "a19d0f50a6512e2f1263a53e4ce4e7c4"
server_key_iv: "e5c9ea27b46ff8e533ffc95b61cfaef9"
server_key_version: "encryptionkey@0"
I20260926 12:34:20.467891 20711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:34:20.472808 20711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:34:20.473536 20711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:34:20.473721 20711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:34:20.474298 20711 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:34:20.475245 20711 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:34:20.475293 20711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:20.475345 20711 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:34:20.475374 20711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:20.484591 20711 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:43063
I20260926 12:34:20.484643 20838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:43063 every 8 connection(s)
I20260926 12:34:20.485601 20711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:34:20.489032 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 20711
I20260926 12:34:20.489185 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:34:20.489499 10632 external_mini_cluster.cc:1428] Setting key 8bb7257a8c7b040538498f1466cecdee
I20260926 12:34:20.493237 20428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:20.487594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:43571 (local address 127.10.98.62:41583)
0926 12:34:20.487775 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:20.487780 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:20.489382 (+ 1602us) server_negotiation.cc:407] Connection header received
0926 12:34:20.490958 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:20.490963 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:20.491012 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:20.491103 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:20.492695 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:34:20.492700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:34:20.492703 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:34:20.492720 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:34:20.492741 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:34:20.492751 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:34:20.492836 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:20.493004 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 12:34:20.493068 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 12:34:20.494273 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:20.488136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:41583 (local address 127.10.98.3:43571)
0926 12:34:20.489162 (+ 1026us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:20.489240 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:20.490702 (+ 1462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:20.491449 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:20.491458 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:20.491599 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:20.492273 (+ 674us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:34:20.492275 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:20.492333 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:34:20.492434 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:34:20.492844 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:34:20.492852 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:34:20.492857 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 12:34:20.493014 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 12:34:20.493274 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":130,"threads_started":1}
I20260926 12:34:20.495659 20839 heartbeater.cc:347] Connected to a master server at 127.10.98.62:41583
I20260926 12:34:20.495982 20839 heartbeater.cc:464] Registering TS with master...
I20260926 12:34:20.496483 20839 heartbeater.cc:511] Master 127.10.98.62:41583 requested a full tablet report, sending...
I20260926 12:34:20.497684 20388 ts_manager.cc:194] Registered new tserver with Master: 489e925e648e43c894debfcb185270d5 (127.10.98.3:43063)
I20260926 12:34:20.504627 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 12:34:20.507786 20428 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:34:20.506727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39468 (local address 127.10.98.62:41583)
0926 12:34:20.506903 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:20.506908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:20.506996 (+ 88us) server_negotiation.cc:407] Connection header received
0926 12:34:20.507229 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:20.507232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:20.507271 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:20.507335 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:20.507664 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39468: BlockingRecv error: recv got EOF from 127.0.0.1:39468 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 12:34:20.508249 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 20449
I20260926 12:34:20.513545 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 20580
I20260926 12:34:20.518249 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 20711
I20260926 12:34:20.522742 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 20358
2026-09-26T12:34:20Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (829 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:34:20 dist-test-slave-t7tk krb5kdc[20849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:34:20 dist-test-slave-t7tk krb5kdc[20849](info): set up 2 sockets
Sep 26 12:34:20 dist-test-slave-t7tk krb5kdc[20849](info): commencing operation
krb5kdc: starting...
W20260926 12:34:21.491580 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:34:21 dist-test-slave-t7tk krb5kdc[20849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:34:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:34:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:34:21.620906 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:43687
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:46627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:43687
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:34:21.731781 20862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:34:21.732059 20862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:34:21.732117 20862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:34:21.736198 20862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:34:21.736267 20862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:34:21.736294 20862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:34:21.736313 20862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:34:21.736346 20862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:34:21.741873 20862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:46627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:43687
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:43687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:34:21.742972 20862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:34:21.743970 20862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:34:21.752552 20870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:21.752556 20868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:21.752557 20867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:21.753314 20862 server_base.cc:1034] running on GCE node
I20260926 12:34:21.753913 20862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:34:21.755072 20862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:34:21.756250 20862 hybrid_clock.cc:648] HybridClock initialized: now 1790426061756233 us; error 34 us; skew 500 ppm
Sep 26 12:34:21 dist-test-slave-t7tk krb5kdc[20849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426061, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:34:21.767422 20862 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:34:21.768474 20862 webserver.cc:460] Webserver started at http://127.10.98.62:42835/ using document root <none> and password file <none>
I20260926 12:34:21.769070 20862 fs_manager.cc:359] Metadata directory not provided
I20260926 12:34:21.769188 20862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:34:21.769382 20862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:34:21.771498 20862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "9b4e69e9d38f4ffeaf59a7076ea386e8"
format_stamp: "Formatted at 2026-09-26 12:34:21 on dist-test-slave-t7tk"
server_key: "f58fc9cb2dd7041c59c4d33bce5386fb"
server_key_iv: "54af1e7097fc154d3ab1de05138180f8"
server_key_version: "encryptionkey@0"
I20260926 12:34:21.772007 20862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "9b4e69e9d38f4ffeaf59a7076ea386e8"
format_stamp: "Formatted at 2026-09-26 12:34:21 on dist-test-slave-t7tk"
server_key: "f58fc9cb2dd7041c59c4d33bce5386fb"
server_key_iv: "54af1e7097fc154d3ab1de05138180f8"
server_key_version: "encryptionkey@0"
I20260926 12:34:21.775893 20862 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:34:21.778648 20877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:21.780299 20862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:34:21.780444 20862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "9b4e69e9d38f4ffeaf59a7076ea386e8"
format_stamp: "Formatted at 2026-09-26 12:34:21 on dist-test-slave-t7tk"
server_key: "f58fc9cb2dd7041c59c4d33bce5386fb"
server_key_iv: "54af1e7097fc154d3ab1de05138180f8"
server_key_version: "encryptionkey@0"
I20260926 12:34:21.780572 20862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:34:21.792768 20862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:34:21.795990 20862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:34:21.796175 20862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:34:21.803202 20862 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:43687
I20260926 12:34:21.803462 20929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:43687 every 8 connection(s)
I20260926 12:34:21.804198 20862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:34:21.807057 20930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:34:21.807380 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 20862
I20260926 12:34:21.807498 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:34:21.807762 10632 external_mini_cluster.cc:1428] Setting key dfa5e3e107fd2e3673eef911e479acd1
I20260926 12:34:21.814283 20930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8: Bootstrap starting.
I20260926 12:34:21.816789 20930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8: Neither blocks nor log segments found. Creating new log.
Sep 26 12:34:21 dist-test-slave-t7tk krb5kdc[20849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:34:21.817571 20930 log.cc:824] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8: Log is configured to *not* fsync() on all Append() calls
I20260926 12:34:21.819823 20930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8: No bootstrap required, opened a new log
I20260926 12:34:21.822875 20933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:21.809212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48030 (local address 127.10.98.62:43687)
0926 12:34:21.809644 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:21.809664 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:21.809708 (+ 44us) server_negotiation.cc:407] Connection header received
0926 12:34:21.810421 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:21.810445 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:21.810873 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:21.811262 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:21.811917 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:21.813460 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:21.814356 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:21.815033 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:21.815185 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:21.817703 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:34:21.817709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:34:21.817725 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:34:21.817758 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:34:21.819643 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:34:21.820095 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:34:21.820100 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:34:21.820107 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:34:21.820254 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:34:21.820592 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:34:21.820595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:34:21.820597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:34:21.821164 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:34:21.821372 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:21.821810 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 12:34:21.822132 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":132,"threads_started":1}
I20260926 12:34:21.824021 20930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4e69e9d38f4ffeaf59a7076ea386e8" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43687 } }
I20260926 12:34:21.824304 20930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:34:21.824352 20930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b4e69e9d38f4ffeaf59a7076ea386e8, State: Initialized, Role: FOLLOWER
I20260926 12:34:21.824828 20930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4e69e9d38f4ffeaf59a7076ea386e8" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43687 } }
I20260926 12:34:21.824980 20930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:34:21.825042 20930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:34:21.825156 20930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:34:21.826062 20930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4e69e9d38f4ffeaf59a7076ea386e8" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43687 } }
I20260926 12:34:21.826383 20930 leader_election.cc:302] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b4e69e9d38f4ffeaf59a7076ea386e8; no voters:
I20260926 12:34:21.826694 20930 leader_election.cc:288] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:34:21.826835 20935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:34:21.827061 20935 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8 [term 1 LEADER]: Becoming Leader. State: Replica: 9b4e69e9d38f4ffeaf59a7076ea386e8, State: Running, Role: LEADER
I20260926 12:34:21.827394 20935 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4e69e9d38f4ffeaf59a7076ea386e8" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43687 } }
I20260926 12:34:21.827872 20930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:34:21.828953 20937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b4e69e9d38f4ffeaf59a7076ea386e8. Latest consensus state: current_term: 1 leader_uuid: "9b4e69e9d38f4ffeaf59a7076ea386e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4e69e9d38f4ffeaf59a7076ea386e8" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43687 } } }
I20260926 12:34:21.829059 20937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8 [sys.catalog]: This master's current role is: LEADER
I20260926 12:34:21.829392 20936 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b4e69e9d38f4ffeaf59a7076ea386e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4e69e9d38f4ffeaf59a7076ea386e8" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43687 } } }
I20260926 12:34:21.829481 20936 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8 [sys.catalog]: This master's current role is: LEADER
I20260926 12:34:21.829569 20944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:34:21.832690 20944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:34:21.838459 20944 catalog_manager.cc:1347] Generated new cluster ID: 9ac871032fee4b30b7cc71866556c5ef
I20260926 12:34:21.838553 20944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:34:21.848901 20944 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:34:21.850251 20944 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:34:21.862988 20944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8: Generated new TSK 0
I20260926 12:34:21.863885 20944 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
I20260926 12:34:21.897599 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:43687
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:46627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:34:22.011314 20956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:34:22.011608 20956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:34:22.011655 20956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:34:22.015753 20956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:34:22.015829 20956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:34:22.015892 20956 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:34:22.021282 20956 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:46627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:43687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20956
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:34:22.022406 20956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:34:22.023496 20956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:34:22.033406 20964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:22.033396 20961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:22.033555 20956 server_base.cc:1034] running on GCE node
W20260926 12:34:22.033396 20962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:22.034153 20956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:34:22.034819 20956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:34:22.036006 20956 hybrid_clock.cc:648] HybridClock initialized: now 1790426062035985 us; error 33 us; skew 500 ppm
Sep 26 12:34:22 dist-test-slave-t7tk krb5kdc[20849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426062, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:34:22.039125 20956 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:34:22.040088 20956 webserver.cc:460] Webserver started at http://127.10.98.1:36409/ using document root <none> and password file <none>
I20260926 12:34:22.040673 20956 fs_manager.cc:359] Metadata directory not provided
I20260926 12:34:22.040831 20956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:34:22.041047 20956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:34:22.043290 20956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "6752e571eea34ff597b651ecae5501b6"
format_stamp: "Formatted at 2026-09-26 12:34:22 on dist-test-slave-t7tk"
server_key: "9c4ebc41a52b22cbf542bcb3c3d083fa"
server_key_iv: "f51953d10b195e53f7b37eadb4a08ba9"
server_key_version: "encryptionkey@0"
I20260926 12:34:22.043840 20956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "6752e571eea34ff597b651ecae5501b6"
format_stamp: "Formatted at 2026-09-26 12:34:22 on dist-test-slave-t7tk"
server_key: "9c4ebc41a52b22cbf542bcb3c3d083fa"
server_key_iv: "f51953d10b195e53f7b37eadb4a08ba9"
server_key_version: "encryptionkey@0"
I20260926 12:34:22.047662 20956 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260926 12:34:22.050382 20971 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:22.051760 20956 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:34:22.051890 20956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "6752e571eea34ff597b651ecae5501b6"
format_stamp: "Formatted at 2026-09-26 12:34:22 on dist-test-slave-t7tk"
server_key: "9c4ebc41a52b22cbf542bcb3c3d083fa"
server_key_iv: "f51953d10b195e53f7b37eadb4a08ba9"
server_key_version: "encryptionkey@0"
I20260926 12:34:22.051997 20956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:34:22.064916 20956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:34:22.068151 20956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:34:22.068338 20956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:34:22.068931 20956 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:34:22.069900 20956 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:34:22.069952 20956 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:22.070004 20956 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:34:22.070030 20956 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:22.078706 20956 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:43433
I20260926 12:34:22.078733 21084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:43433 every 8 connection(s)
I20260926 12:34:22.079617 20956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:34:22.084062 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 20956
I20260926 12:34:22.084203 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:34:22.084491 10632 external_mini_cluster.cc:1428] Setting key b664966b8f0108e1df689699e9faa9d0
Sep 26 12:34:22 dist-test-slave-t7tk krb5kdc[20849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426062, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.98.2@KRBTEST.COM; defaulting to no policy
I20260926 12:34:22.095664 20933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:22.081775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:40055 (local address 127.10.98.62:43687)
0926 12:34:22.081906 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:22.081910 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:22.082693 (+ 783us) server_negotiation.cc:407] Connection header received
0926 12:34:22.084184 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:22.084187 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:22.084250 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:22.084329 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:22.085900 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.086466 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.088110 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.088490 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.088737 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:22.091488 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:34:22.091493 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:34:22.091496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:34:22.091530 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:34:22.093239 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:34:22.093915 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:34:22.093920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:34:22.093922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:34:22.094022 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:34:22.094368 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:34:22.094371 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:34:22.094372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:34:22.094631 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:34:22.094735 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:22.095387 (+ 652us) server_negotiation.cc:299] Negotiation successful
0926 12:34:22.095524 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:34:22.096681 21087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:22.082042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:43687 (local address 127.10.98.1:40055)
0926 12:34:22.082523 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:22.082571 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:22.083914 (+ 1343us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:22.084537 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:22.084548 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:22.084944 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:22.085697 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:22.085717 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.086613 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.086617 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:22.087963 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:22.087975 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.088783 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.088787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:22.088943 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:22.089595 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:22.089613 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:22.091285 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:34:22.093378 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:34:22.093384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:34:22.093399 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:34:22.093801 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:34:22.094117 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:34:22.094119 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:34:22.094120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:34:22.094279 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:34:22.094766 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:34:22.094772 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:34:22.095102 (+ 330us) client_negotiation.cc:769] Sending connection context
0926 12:34:22.095310 (+ 208us) client_negotiation.cc:240] Negotiation successful
0926 12:34:22.095685 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":123,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.2@KRBTEST.COM" created.
I20260926 12:34:22.098097 21085 heartbeater.cc:347] Connected to a master server at 127.10.98.62:43687
I20260926 12:34:22.098346 21085 heartbeater.cc:464] Registering TS with master...
I20260926 12:34:22.098860 21085 heartbeater.cc:511] Master 127.10.98.62:43687 requested a full tablet report, sending...
I20260926 12:34:22.100471 20894 ts_manager.cc:194] Registered new tserver with Master: 6752e571eea34ff597b651ecae5501b6 (127.10.98.1:43433)
I20260926 12:34:22.101814 20894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:40055
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
I20260926 12:34:22.116679 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:43687
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:46627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:34:22.229334 21090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:34:22.229619 21090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:34:22.229688 21090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:34:22.233933 21090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:34:22.234004 21090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:34:22.234095 21090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:34:22.239526 21090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:46627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:43687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21090
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:34:22.240641 21090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:34:22.241747 21090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:34:22.251809 21098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:22.251817 21095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:22.251827 21096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:22.252122 21090 server_base.cc:1034] running on GCE node
I20260926 12:34:22.252521 21090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:34:22.253209 21090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:34:22.254396 21090 hybrid_clock.cc:648] HybridClock initialized: now 1790426062254380 us; error 28 us; skew 500 ppm
Sep 26 12:34:22 dist-test-slave-t7tk krb5kdc[20849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426062, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:34:22.257604 21090 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:34:22.258742 21090 webserver.cc:460] Webserver started at http://127.10.98.2:33359/ using document root <none> and password file <none>
I20260926 12:34:22.259316 21090 fs_manager.cc:359] Metadata directory not provided
I20260926 12:34:22.259384 21090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:34:22.259578 21090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:34:22.261775 21090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "8d91235f97ce4000afb4ad67dc4b3764"
format_stamp: "Formatted at 2026-09-26 12:34:22 on dist-test-slave-t7tk"
server_key: "89d776465cf1211e23cd985220c6be84"
server_key_iv: "4d1b110affa3d268b854f7a2ed0a6400"
server_key_version: "encryptionkey@0"
I20260926 12:34:22.262336 21090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "8d91235f97ce4000afb4ad67dc4b3764"
format_stamp: "Formatted at 2026-09-26 12:34:22 on dist-test-slave-t7tk"
server_key: "89d776465cf1211e23cd985220c6be84"
server_key_iv: "4d1b110affa3d268b854f7a2ed0a6400"
server_key_version: "encryptionkey@0"
I20260926 12:34:22.266237 21090 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:34:22.268703 21105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:22.269994 21090 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:34:22.270110 21090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "8d91235f97ce4000afb4ad67dc4b3764"
format_stamp: "Formatted at 2026-09-26 12:34:22 on dist-test-slave-t7tk"
server_key: "89d776465cf1211e23cd985220c6be84"
server_key_iv: "4d1b110affa3d268b854f7a2ed0a6400"
server_key_version: "encryptionkey@0"
I20260926 12:34:22.270226 21090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:34:22.287529 21090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:34:22.290692 21090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:34:22.290871 21090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:34:22.291456 21090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:34:22.292272 21090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:34:22.292320 21090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:22.292372 21090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:34:22.292400 21090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:22.301520 21090 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:40195
I20260926 12:34:22.301550 21218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:40195 every 8 connection(s)
I20260926 12:34:22.302450 21090 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:34:22.303311 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 21090
I20260926 12:34:22.303436 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:34:22.303756 10632 external_mini_cluster.cc:1428] Setting key a3fd5c6c76db0b3409e7b2780aec94ae
WARNING: no policy specified for kudu/127.10.98.3@KRBTEST.COM; defaulting to no policy
Sep 26 12:34:22 dist-test-slave-t7tk krb5kdc[20849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426062, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.3@KRBTEST.COM" created.
I20260926 12:34:22.317317 20933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:22.304389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:44163 (local address 127.10.98.62:43687)
0926 12:34:22.304511 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:22.304515 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:22.305661 (+ 1146us) server_negotiation.cc:407] Connection header received
0926 12:34:22.307117 (+ 1456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:22.307121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:22.307187 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:22.307265 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:22.308447 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.309274 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.310577 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.310922 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.311034 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:22.313584 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:34:22.313588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:34:22.313590 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:34:22.313619 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:34:22.315079 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:34:22.315619 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:34:22.315622 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:34:22.315623 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:34:22.315715 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:34:22.316100 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:34:22.316103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:34:22.316104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:34:22.316322 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:34:22.316412 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:22.316977 (+ 565us) server_negotiation.cc:299] Negotiation successful
0926 12:34:22.317152 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 12:34:22.318209 21221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:22.304982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:43687 (local address 127.10.98.2:44163)
0926 12:34:22.305510 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:22.305568 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:22.306878 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:22.307409 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:22.307416 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:22.307787 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:22.308294 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:22.308308 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.309411 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.309414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:22.310443 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:22.310453 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.311063 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.311065 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:22.311219 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:22.311820 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:22.311836 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:22.313393 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:34:22.315216 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:34:22.315222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:34:22.315236 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:34:22.315500 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:34:22.315815 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:34:22.315818 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:34:22.315819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:34:22.315965 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:34:22.316421 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:34:22.316428 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:34:22.316742 (+ 314us) client_negotiation.cc:769] Sending connection context
0926 12:34:22.316915 (+ 173us) client_negotiation.cc:240] Negotiation successful
0926 12:34:22.317282 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":121,"threads_started":1}
I20260926 12:34:22.319494 21219 heartbeater.cc:347] Connected to a master server at 127.10.98.62:43687
I20260926 12:34:22.319787 21219 heartbeater.cc:464] Registering TS with master...
I20260926 12:34:22.320261 21219 heartbeater.cc:511] Master 127.10.98.62:43687 requested a full tablet report, sending...
I20260926 12:34:22.321331 20894 ts_manager.cc:194] Registered new tserver with Master: 8d91235f97ce4000afb4ad67dc4b3764 (127.10.98.2:40195)
I20260926 12:34:22.322094 20894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:44163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
I20260926 12:34:22.332945 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:43687
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:46627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:34:22.448429 21224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:34:22.448710 21224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:34:22.448764 21224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:34:22.452998 21224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:34:22.453080 21224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:34:22.453184 21224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:34:22.458815 21224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:46627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:43687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:34:22.459923 21224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:34:22.460938 21224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:34:22.470896 21232 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:22.470877 21230 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:34:22.470901 21229 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:34:22.471314 21224 server_base.cc:1034] running on GCE node
I20260926 12:34:22.471732 21224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:34:22.472355 21224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:34:22.473536 21224 hybrid_clock.cc:648] HybridClock initialized: now 1790426062473470 us; error 78 us; skew 500 ppm
Sep 26 12:34:22 dist-test-slave-t7tk krb5kdc[20849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426062, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:34:22.476434 21224 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:34:22.477476 21224 webserver.cc:460] Webserver started at http://127.10.98.3:35861/ using document root <none> and password file <none>
I20260926 12:34:22.478015 21224 fs_manager.cc:359] Metadata directory not provided
I20260926 12:34:22.478082 21224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:34:22.478281 21224 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:34:22.480430 21224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "12aa34ffa5324fa9b54bc9629d07d417"
format_stamp: "Formatted at 2026-09-26 12:34:22 on dist-test-slave-t7tk"
server_key: "f680a9f54ff67772a3305545f541e5f2"
server_key_iv: "6def3b539ac53c6bfa27995070a2a5a7"
server_key_version: "encryptionkey@0"
I20260926 12:34:22.481006 21224 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "12aa34ffa5324fa9b54bc9629d07d417"
format_stamp: "Formatted at 2026-09-26 12:34:22 on dist-test-slave-t7tk"
server_key: "f680a9f54ff67772a3305545f541e5f2"
server_key_iv: "6def3b539ac53c6bfa27995070a2a5a7"
server_key_version: "encryptionkey@0"
I20260926 12:34:22.485069 21224 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 12:34:22.487867 21239 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:22.489454 21224 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:34:22.489569 21224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "12aa34ffa5324fa9b54bc9629d07d417"
format_stamp: "Formatted at 2026-09-26 12:34:22 on dist-test-slave-t7tk"
server_key: "f680a9f54ff67772a3305545f541e5f2"
server_key_iv: "6def3b539ac53c6bfa27995070a2a5a7"
server_key_version: "encryptionkey@0"
I20260926 12:34:22.489666 21224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:34:22.509495 21224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:34:22.512583 21224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:34:22.512751 21224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:34:22.513350 21224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:34:22.514169 21224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:34:22.514217 21224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:22.514259 21224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:34:22.514274 21224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:34:22.522792 21224 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:39133
I20260926 12:34:22.522856 21352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:39133 every 8 connection(s)
I20260926 12:34:22.523687 21224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:34:22.529671 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 21224
I20260926 12:34:22.529839 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:34:22.530141 10632 external_mini_cluster.cc:1428] Setting key dcaa83df65dc5d58891a7f6fdf6bcfd8
Sep 26 12:34:22 dist-test-slave-t7tk krb5kdc[20849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426062, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:34:22.538192 20933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:22.525527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:43311 (local address 127.10.98.62:43687)
0926 12:34:22.525647 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:22.525650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:22.526716 (+ 1066us) server_negotiation.cc:407] Connection header received
0926 12:34:22.528190 (+ 1474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:22.528194 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:22.528258 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:22.528335 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:22.529691 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.530305 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.532023 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.532240 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.532328 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:22.534903 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:34:22.534907 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:34:22.534909 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:34:22.534936 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:34:22.536189 (+ 1253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:34:22.536671 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:34:22.536675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:34:22.536677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:34:22.536757 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:34:22.537070 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:34:22.537073 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:34:22.537075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:34:22.537295 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:34:22.537393 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:22.537892 (+ 499us) server_negotiation.cc:299] Negotiation successful
0926 12:34:22.538019 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 12:34:22.538995 21355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:22.525964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:43687 (local address 127.10.98.3:43311)
0926 12:34:22.526479 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:22.526590 (+ 111us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:22.527951 (+ 1361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:22.528570 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:22.528577 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:22.528947 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:34:22.529537 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:22.529553 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.530429 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.530432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:22.531408 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:22.531418 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.532389 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.532394 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:22.532572 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:22.533091 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:34:22.533133 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:34:22.534729 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:34:22.536301 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:34:22.536307 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:34:22.536324 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:34:22.536577 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:34:22.536837 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:34:22.536840 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:34:22.536842 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:34:22.536983 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:34:22.537393 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:34:22.537400 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:34:22.537660 (+ 260us) client_negotiation.cc:769] Sending connection context
0926 12:34:22.537836 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 12:34:22.538173 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":123,"threads_started":1}
I20260926 12:34:22.540162 21353 heartbeater.cc:347] Connected to a master server at 127.10.98.62:43687
I20260926 12:34:22.540385 21353 heartbeater.cc:464] Registering TS with master...
I20260926 12:34:22.540858 21353 heartbeater.cc:511] Master 127.10.98.62:43687 requested a full tablet report, sending...
I20260926 12:34:22.542021 20894 ts_manager.cc:194] Registered new tserver with Master: 12aa34ffa5324fa9b54bc9629d07d417 (127.10.98.3:39133)
I20260926 12:34:22.542735 20894 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:43311
I20260926 12:34:22.544119 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:34:22.552904 20933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:22.546008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48034 (local address 127.10.98.62:43687)
0926 12:34:22.546123 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:22.546127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:22.546391 (+ 264us) server_negotiation.cc:407] Connection header received
0926 12:34:22.546468 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:22.546470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:22.546512 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:22.546568 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:34:22.547211 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.547745 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.548623 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.548842 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.548926 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:22.550162 (+ 1236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:34:22.550166 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:34:22.550168 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:34:22.550191 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:34:22.551503 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:34:22.551924 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:34:22.551927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:34:22.551929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:34:22.552007 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:34:22.552191 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:34:22.552193 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:34:22.552194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:34:22.552424 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:34:22.552521 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:22.552676 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 12:34:22.552769 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:34:22.556107 20893 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48034:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:34:22.559090 20893 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:34:22.573256 21366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:22.567292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:40195 (local address 127.0.0.1:43060)
0926 12:34:22.568101 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:22.568150 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:22.568293 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:22.569188 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:22.569191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:22.569209 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:34:22.569349 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:22.569356 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.570404 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.570407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:22.571694 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:22.571705 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.572636 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.572639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:22.572761 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:22.572801 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 12:34:22.572912 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 12:34:22.573040 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"mutex_wait_us":24,"spinlock_wait_cycles":896,"thread_start_us":100,"threads_started":1}
I20260926 12:34:22.573625 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:22.567457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43060 (local address 127.10.98.2:40195)
0926 12:34:22.568704 (+ 1247us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:22.568717 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:22.568740 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:34:22.568812 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:22.568817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:22.569029 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:22.569213 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:34:22.569500 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.570284 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.571834 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.572514 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.572624 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:22.572800 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:22.573372 (+ 572us) server_negotiation.cc:299] Negotiation successful
0926 12:34:22.573475 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1138,"thread_start_us":93,"threads_started":1}
I20260926 12:34:22.573642 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:22.567597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:43433 (local address 127.0.0.1:45794)
0926 12:34:22.568100 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:22.568150 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:22.568282 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:22.569082 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:22.569088 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:22.569136 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:34:22.569298 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:22.569305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.570781 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.570785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:22.571988 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:22.571997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.573207 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.573210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:22.573350 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:22.573374 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:34:22.573444 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 12:34:22.573506 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"mutex_wait_us":61,"spinlock_wait_cycles":9600,"thread_start_us":101,"threads_started":1}
I20260926 12:34:22.573699 21368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:22.568034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:39133 (local address 127.0.0.1:57982)
0926 12:34:22.568242 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:22.568254 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:22.568324 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:22.569436 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:22.569439 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:22.569457 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:34:22.569537 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:22.569544 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.570803 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.570807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:22.571862 (+ 1055us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:22.571871 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.572942 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.572946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:22.573025 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:22.573045 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 12:34:22.573133 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 12:34:22.573196 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":25,"thread_start_us":89,"threads_started":1}
I20260926 12:34:22.574023 21369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:22.568452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57982 (local address 127.10.98.3:39133)
0926 12:34:22.568994 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:22.569005 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:22.569025 (+ 20us) server_negotiation.cc:407] Connection header received
0926 12:34:22.569120 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:22.569125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:22.569308 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:22.569449 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:34:22.569623 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.570698 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.571972 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.572822 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.572935 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:22.573081 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:22.573176 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 12:34:22.573284 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":154,"threads_started":1}
I20260926 12:34:22.574133 21367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:22.567750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45794 (local address 127.10.98.1:43433)
0926 12:34:22.568066 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:22.568082 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:22.568418 (+ 336us) server_negotiation.cc:407] Connection header received
0926 12:34:22.568492 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:22.568498 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:22.568728 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:22.568888 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:34:22.569433 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.570622 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.572104 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.572864 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.573002 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:22.573149 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:22.573884 (+ 735us) server_negotiation.cc:299] Negotiation successful
0926 12:34:22.573984 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":57,"threads_started":1}
I20260926 12:34:22.576918 21287 tablet_service.cc:1467] Processing CreateTablet for tablet 3612cf22dde74a1eb4c5357f3f390bb1 (DEFAULT_TABLE table=test-table [id=f4002dd8d8d04e94a4ea37358d69313f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:34:22.576918 21153 tablet_service.cc:1467] Processing CreateTablet for tablet 3612cf22dde74a1eb4c5357f3f390bb1 (DEFAULT_TABLE table=test-table [id=f4002dd8d8d04e94a4ea37358d69313f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:34:22.578166 21287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3612cf22dde74a1eb4c5357f3f390bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:34:22.578161 21153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3612cf22dde74a1eb4c5357f3f390bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:34:22.580358 21019 tablet_service.cc:1467] Processing CreateTablet for tablet 3612cf22dde74a1eb4c5357f3f390bb1 (DEFAULT_TABLE table=test-table [id=f4002dd8d8d04e94a4ea37358d69313f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:34:22.581687 21019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3612cf22dde74a1eb4c5357f3f390bb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:34:22.586546 21370 tablet_bootstrap.cc:492] T 3612cf22dde74a1eb4c5357f3f390bb1 P 8d91235f97ce4000afb4ad67dc4b3764: Bootstrap starting.
I20260926 12:34:22.586750 21371 tablet_bootstrap.cc:492] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417: Bootstrap starting.
I20260926 12:34:22.588261 21372 tablet_bootstrap.cc:492] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6: Bootstrap starting.
I20260926 12:34:22.588863 21371 tablet_bootstrap.cc:654] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417: Neither blocks nor log segments found. Creating new log.
I20260926 12:34:22.589411 21370 tablet_bootstrap.cc:654] T 3612cf22dde74a1eb4c5357f3f390bb1 P 8d91235f97ce4000afb4ad67dc4b3764: Neither blocks nor log segments found. Creating new log.
I20260926 12:34:22.590106 21370 log.cc:824] T 3612cf22dde74a1eb4c5357f3f390bb1 P 8d91235f97ce4000afb4ad67dc4b3764: Log is configured to *not* fsync() on all Append() calls
I20260926 12:34:22.590209 21371 log.cc:824] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417: Log is configured to *not* fsync() on all Append() calls
I20260926 12:34:22.590499 21372 tablet_bootstrap.cc:654] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6: Neither blocks nor log segments found. Creating new log.
I20260926 12:34:22.591696 21372 log.cc:824] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6: Log is configured to *not* fsync() on all Append() calls
I20260926 12:34:22.592200 21371 tablet_bootstrap.cc:492] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417: No bootstrap required, opened a new log
I20260926 12:34:22.592406 21371 ts_tablet_manager.cc:1397] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260926 12:34:22.592470 21370 tablet_bootstrap.cc:492] T 3612cf22dde74a1eb4c5357f3f390bb1 P 8d91235f97ce4000afb4ad67dc4b3764: No bootstrap required, opened a new log
I20260926 12:34:22.592644 21370 ts_tablet_manager.cc:1397] T 3612cf22dde74a1eb4c5357f3f390bb1 P 8d91235f97ce4000afb4ad67dc4b3764: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:34:22.593569 21372 tablet_bootstrap.cc:492] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6: No bootstrap required, opened a new log
I20260926 12:34:22.593755 21372 ts_tablet_manager.cc:1397] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260926 12:34:22.596402 21371 raft_consensus.cc:348] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d91235f97ce4000afb4ad67dc4b3764" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40195 } } peers { permanent_uuid: "12aa34ffa5324fa9b54bc9629d07d417" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 39133 } } peers { permanent_uuid: "6752e571eea34ff597b651ecae5501b6" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 43433 } }
I20260926 12:34:22.596402 21370 raft_consensus.cc:348] T 3612cf22dde74a1eb4c5357f3f390bb1 P 8d91235f97ce4000afb4ad67dc4b3764 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d91235f97ce4000afb4ad67dc4b3764" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40195 } } peers { permanent_uuid: "12aa34ffa5324fa9b54bc9629d07d417" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 39133 } } peers { permanent_uuid: "6752e571eea34ff597b651ecae5501b6" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 43433 } }
I20260926 12:34:22.596506 21372 raft_consensus.cc:348] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d91235f97ce4000afb4ad67dc4b3764" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40195 } } peers { permanent_uuid: "12aa34ffa5324fa9b54bc9629d07d417" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 39133 } } peers { permanent_uuid: "6752e571eea34ff597b651ecae5501b6" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 43433 } }
I20260926 12:34:22.596686 21370 raft_consensus.cc:374] T 3612cf22dde74a1eb4c5357f3f390bb1 P 8d91235f97ce4000afb4ad67dc4b3764 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:34:22.596686 21371 raft_consensus.cc:374] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:34:22.596704 21372 raft_consensus.cc:374] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:34:22.596745 21370 raft_consensus.cc:729] T 3612cf22dde74a1eb4c5357f3f390bb1 P 8d91235f97ce4000afb4ad67dc4b3764 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d91235f97ce4000afb4ad67dc4b3764, State: Initialized, Role: FOLLOWER
I20260926 12:34:22.596752 21371 raft_consensus.cc:729] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12aa34ffa5324fa9b54bc9629d07d417, State: Initialized, Role: FOLLOWER
I20260926 12:34:22.596765 21372 raft_consensus.cc:729] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6752e571eea34ff597b651ecae5501b6, State: Initialized, Role: FOLLOWER
I20260926 12:34:22.597229 21372 consensus_queue.cc:260] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d91235f97ce4000afb4ad67dc4b3764" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40195 } } peers { permanent_uuid: "12aa34ffa5324fa9b54bc9629d07d417" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 39133 } } peers { permanent_uuid: "6752e571eea34ff597b651ecae5501b6" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 43433 } }
I20260926 12:34:22.597236 21371 consensus_queue.cc:260] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d91235f97ce4000afb4ad67dc4b3764" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40195 } } peers { permanent_uuid: "12aa34ffa5324fa9b54bc9629d07d417" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 39133 } } peers { permanent_uuid: "6752e571eea34ff597b651ecae5501b6" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 43433 } }
I20260926 12:34:22.597236 21370 consensus_queue.cc:260] T 3612cf22dde74a1eb4c5357f3f390bb1 P 8d91235f97ce4000afb4ad67dc4b3764 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d91235f97ce4000afb4ad67dc4b3764" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40195 } } peers { permanent_uuid: "12aa34ffa5324fa9b54bc9629d07d417" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 39133 } } peers { permanent_uuid: "6752e571eea34ff597b651ecae5501b6" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 43433 } }
I20260926 12:34:22.598106 21353 heartbeater.cc:503] Master 127.10.98.62:43687 was elected leader, sending a full tablet report...
I20260926 12:34:22.598232 21372 ts_tablet_manager.cc:1428] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 12:34:22.598232 21371 ts_tablet_manager.cc:1428] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 12:34:22.598232 21370 ts_tablet_manager.cc:1428] T 3612cf22dde74a1eb4c5357f3f390bb1 P 8d91235f97ce4000afb4ad67dc4b3764: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:34:22.598554 21085 heartbeater.cc:503] Master 127.10.98.62:43687 was elected leader, sending a full tablet report...
I20260926 12:34:22.599071 21219 heartbeater.cc:503] Master 127.10.98.62:43687 was elected leader, sending a full tablet report...
W20260926 12:34:22.774693 21354 tablet.cc:2369] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:34:22.803534 21220 tablet.cc:2369] T 3612cf22dde74a1eb4c5357f3f390bb1 P 8d91235f97ce4000afb4ad67dc4b3764: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:34:22.830930 21086 tablet.cc:2369] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:34:22.909482 21376 raft_consensus.cc:482] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:34:22.909732 21376 raft_consensus.cc:504] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d91235f97ce4000afb4ad67dc4b3764" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40195 } } peers { permanent_uuid: "12aa34ffa5324fa9b54bc9629d07d417" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 39133 } } peers { permanent_uuid: "6752e571eea34ff597b651ecae5501b6" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 43433 } }
I20260926 12:34:22.910894 21376 leader_election.cc:288] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d91235f97ce4000afb4ad67dc4b3764 (127.10.98.2:40195), 12aa34ffa5324fa9b54bc9629d07d417 (127.10.98.3:39133)
I20260926 12:34:22.916442 21365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:22.911289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:35405 (local address 127.10.98.2:40195)
0926 12:34:22.911435 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:22.911440 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:22.911673 (+ 233us) server_negotiation.cc:407] Connection header received
0926 12:34:22.911926 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:22.911930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:22.911985 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:22.912060 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:34:22.912477 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.913306 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.915149 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.915756 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.915873 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:22.915945 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:22.916231 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 12:34:22.916315 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:34:22.916442 21369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:22.911776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:56837 (local address 127.10.98.3:39133)
0926 12:34:22.911915 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:22.911920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:22.912074 (+ 154us) server_negotiation.cc:407] Connection header received
0926 12:34:22.912190 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:22.912193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:22.912238 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:22.912309 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:34:22.912585 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.913335 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.915149 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.915836 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.915919 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:22.915969 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:22.916231 (+ 262us) server_negotiation.cc:299] Negotiation successful
0926 12:34:22.916311 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 12:34:22.916496 21379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:22.911189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:40195 (local address 127.10.98.1:35405)
0926 12:34:22.911551 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:22.911570 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:22.911773 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:22.912216 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:22.912219 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:22.912235 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:34:22.912373 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:22.912381 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.913459 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.913463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:22.914980 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:22.914994 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.915902 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.915907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:22.916013 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:22.916079 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 12:34:22.916165 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 12:34:22.916279 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":74,"thread_start_us":130,"threads_started":1}
I20260926 12:34:22.916500 21380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:22.911697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:39133 (local address 127.10.98.1:56837)
0926 12:34:22.911991 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:22.912006 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:34:22.912104 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:34:22.912342 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:34:22.912345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:34:22.912366 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:34:22.912454 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:22.912460 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.913459 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.913463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:22.914980 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:34:22.914994 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.915943 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.915946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:34:22.916100 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:22.916119 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 12:34:22.916181 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 12:34:22.916281 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"mutex_wait_us":164,"thread_start_us":85,"threads_started":1}
I20260926 12:34:22.917361 21307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3612cf22dde74a1eb4c5357f3f390bb1" candidate_uuid: "6752e571eea34ff597b651ecae5501b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12aa34ffa5324fa9b54bc9629d07d417" is_pre_election: true
I20260926 12:34:22.917371 21173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3612cf22dde74a1eb4c5357f3f390bb1" candidate_uuid: "6752e571eea34ff597b651ecae5501b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d91235f97ce4000afb4ad67dc4b3764" is_pre_election: true
I20260926 12:34:22.917670 21173 raft_consensus.cc:2454] T 3612cf22dde74a1eb4c5357f3f390bb1 P 8d91235f97ce4000afb4ad67dc4b3764 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6752e571eea34ff597b651ecae5501b6 in term 0.
I20260926 12:34:22.917677 21307 raft_consensus.cc:2454] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6752e571eea34ff597b651ecae5501b6 in term 0.
I20260926 12:34:22.918133 20975 leader_election.cc:302] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6752e571eea34ff597b651ecae5501b6, 8d91235f97ce4000afb4ad67dc4b3764; no voters:
I20260926 12:34:22.918423 21376 raft_consensus.cc:2781] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:34:22.918506 21376 raft_consensus.cc:482] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:34:22.918535 21376 raft_consensus.cc:3037] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:34:22.919566 21376 raft_consensus.cc:504] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d91235f97ce4000afb4ad67dc4b3764" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40195 } } peers { permanent_uuid: "12aa34ffa5324fa9b54bc9629d07d417" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 39133 } } peers { permanent_uuid: "6752e571eea34ff597b651ecae5501b6" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 43433 } }
I20260926 12:34:22.919901 21376 leader_election.cc:288] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6 [CANDIDATE]: Term 1 election: Requested vote from peers 8d91235f97ce4000afb4ad67dc4b3764 (127.10.98.2:40195), 12aa34ffa5324fa9b54bc9629d07d417 (127.10.98.3:39133)
I20260926 12:34:22.920303 21173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3612cf22dde74a1eb4c5357f3f390bb1" candidate_uuid: "6752e571eea34ff597b651ecae5501b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d91235f97ce4000afb4ad67dc4b3764"
I20260926 12:34:22.920354 21307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3612cf22dde74a1eb4c5357f3f390bb1" candidate_uuid: "6752e571eea34ff597b651ecae5501b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12aa34ffa5324fa9b54bc9629d07d417"
I20260926 12:34:22.920436 21173 raft_consensus.cc:3037] T 3612cf22dde74a1eb4c5357f3f390bb1 P 8d91235f97ce4000afb4ad67dc4b3764 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:34:22.920483 21307 raft_consensus.cc:3037] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:34:22.921545 21307 raft_consensus.cc:2454] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6752e571eea34ff597b651ecae5501b6 in term 1.
I20260926 12:34:22.921523 21173 raft_consensus.cc:2454] T 3612cf22dde74a1eb4c5357f3f390bb1 P 8d91235f97ce4000afb4ad67dc4b3764 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6752e571eea34ff597b651ecae5501b6 in term 1.
I20260926 12:34:22.921849 20975 leader_election.cc:302] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12aa34ffa5324fa9b54bc9629d07d417, 6752e571eea34ff597b651ecae5501b6; no voters:
I20260926 12:34:22.922035 21376 raft_consensus.cc:2781] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:34:22.922303 21376 raft_consensus.cc:686] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6 [term 1 LEADER]: Becoming Leader. State: Replica: 6752e571eea34ff597b651ecae5501b6, State: Running, Role: LEADER
I20260926 12:34:22.922683 21376 consensus_queue.cc:237] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d91235f97ce4000afb4ad67dc4b3764" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40195 } } peers { permanent_uuid: "12aa34ffa5324fa9b54bc9629d07d417" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 39133 } } peers { permanent_uuid: "6752e571eea34ff597b651ecae5501b6" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 43433 } }
I20260926 12:34:22.926223 20892 catalog_manager.cc:5495] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6752e571eea34ff597b651ecae5501b6 (127.10.98.1). New cstate: current_term: 1 leader_uuid: "6752e571eea34ff597b651ecae5501b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d91235f97ce4000afb4ad67dc4b3764" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12aa34ffa5324fa9b54bc9629d07d417" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 39133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6752e571eea34ff597b651ecae5501b6" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 43433 } health_report { overall_health: HEALTHY } } }
I20260926 12:34:22.941128 21367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:22.937511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45808 (local address 127.10.98.1:43433)
0926 12:34:22.937733 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 12:34:22.937736 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:34:22.937747 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:34:22.937823 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:34:22.937825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:34:22.937888 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:34:22.937985 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:34:22.938241 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.938813 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.939684 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:34:22.939897 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:34:22.939982 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:34:22.940280 (+ 298us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:34:22.940711 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:34:22.940824 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 12:34:22.940888 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 12:34:22.940947 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260926 12:34:22.947295 21173 raft_consensus.cc:1261] T 3612cf22dde74a1eb4c5357f3f390bb1 P 8d91235f97ce4000afb4ad67dc4b3764 [term 1 FOLLOWER]: Refusing update from remote peer 6752e571eea34ff597b651ecae5501b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:34:22.947294 21307 raft_consensus.cc:1261] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417 [term 1 FOLLOWER]: Refusing update from remote peer 6752e571eea34ff597b651ecae5501b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:34:22.948045 21381 consensus_queue.cc:1035] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d91235f97ce4000afb4ad67dc4b3764" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:34:22.948285 21376 consensus_queue.cc:1035] T 3612cf22dde74a1eb4c5357f3f390bb1 P 6752e571eea34ff597b651ecae5501b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12aa34ffa5324fa9b54bc9629d07d417" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 39133 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:34:22.957491 21385 mvcc.cc:205] Tried to move back new op lower bound from 7333585153824579584 to 7333585153734094848. Current Snapshot: MvccSnapshot[applied={T|T < 7333585153824579584}]
I20260926 12:34:22.964443 20893 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:34:22.964607 20893 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48034:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:34:22.967057 20893 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b4e69e9d38f4ffeaf59a7076ea386e8: Sending DeleteTablet for 3 replicas of tablet 3612cf22dde74a1eb4c5357f3f390bb1
I20260926 12:34:22.967936 21153 tablet_service.cc:1514] Processing DeleteTablet for tablet 3612cf22dde74a1eb4c5357f3f390bb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:34:22 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:43060
I20260926 12:34:22.968062 21019 tablet_service.cc:1514] Processing DeleteTablet for tablet 3612cf22dde74a1eb4c5357f3f390bb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:34:22 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:45794
I20260926 12:34:22.968103 21287 tablet_service.cc:1514] Processing DeleteTablet for tablet 3612cf22dde74a1eb4c5357f3f390bb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:34:22 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:57982
I20260926 12:34:22.968500 21394 tablet_replica.cc:330] T 3612cf22dde74a1eb4c5357f3f390bb1 P 8d91235f97ce4000afb4ad67dc4b3764: stopping tablet replica
I20260926 12:34:22.968837 21394 raft_consensus.cc:2229] T 3612cf22dde74a1eb4c5357f3f390bb1 P 8d91235f97ce4000afb4ad67dc4b3764 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:34:22.969214 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 20956
I20260926 12:34:22.971361 21394 raft_consensus.cc:2258] T 3612cf22dde74a1eb4c5357f3f390bb1 P 8d91235f97ce4000afb4ad67dc4b3764 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:34:22.972812 21394 ts_tablet_manager.cc:1905] T 3612cf22dde74a1eb4c5357f3f390bb1 P 8d91235f97ce4000afb4ad67dc4b3764: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:34:22.973331 21396 tablet_replica.cc:330] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417: stopping tablet replica
I20260926 12:34:22.973661 21396 raft_consensus.cc:2229] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:34:22.974092 21396 raft_consensus.cc:2258] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 12:34:22.974912 20878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.98.1:43433 (error 108)
I20260926 12:34:22.975088 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 21090
I20260926 12:34:22.976375 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:22.976078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:43433 (local address 127.0.0.1:45810)
0926 12:34:22.976216 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:22.976287 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.1:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 12:34:22.976553 20878 catalog_manager.cc:4584] TS 6752e571eea34ff597b651ecae5501b6 (127.10.98.1:43433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3612cf22dde74a1eb4c5357f3f390bb1: Network error: Client connection negotiation failed: client connection to 127.10.98.1:43433: connect: Connection refused (error 111)
I20260926 12:34:22.977836 21396 ts_tablet_manager.cc:1905] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:34:22.980094 21396 ts_tablet_manager.cc:1918] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:34:22.980187 21396 log.cc:1196] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/3612cf22dde74a1eb4c5357f3f390bb1
I20260926 12:34:22.980487 21396 ts_tablet_manager.cc:1939] T 3612cf22dde74a1eb4c5357f3f390bb1 P 12aa34ffa5324fa9b54bc9629d07d417: Deleting consensus metadata
I20260926 12:34:22.980775 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 21224
I20260926 12:34:22.981585 21364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:34:22.981046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:40195 (local address 127.0.0.1:43072)
0926 12:34:22.981461 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0926 12:34:22.981508 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.2:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339}
I20260926 12:34:22.986122 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 20862
2026-09-26T12:34:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2465 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:34:23 dist-test-slave-t7tk krb5kdc[21400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:34:23 dist-test-slave-t7tk krb5kdc[21400](info): set up 2 sockets
Sep 26 12:34:23 dist-test-slave-t7tk krb5kdc[21400](info): commencing operation
krb5kdc: starting...
W20260926 12:34:23.967528 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:34:24 dist-test-slave-t7tk krb5kdc[21400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:34:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:34:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/rangeradmin_127.10.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/rangeradmin_127.10.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/rangerlookup_127.10.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/rangerlookup_127.10.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/HTTP_127.10.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/HTTP_127.10.98.20@KRBTEST.COM.keytab.
I20260926 12:34:24.140280 10632 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskFMA2CE/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/postgres -l logfile start
2026-09-26 12:34:26.865 UTC [21430] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 12:34:26.865 UTC [21430] LOG: listening on IPv4 address "127.10.98.20", port 41321
2026-09-26 12:34:26.869 UTC [21430] LOG: listening on Unix socket "/tmp/.s.PGSQL.41321"
2026-09-26 12:34:26.898 UTC [21433] LOG: database system was shut down at 2026-09-26 12:34:25 UTC
2026-09-26 12:34:26.901 UTC [21430] LOG: database system is ready to accept connections
I20260926 12:34:27.775131 10632 mini_postgres.cc:96] Postgres bound to 41321
2026-09-26 12:34:27.781 UTC [21441] FATAL: database "slave" does not exist
127.10.98.20:41321 - accepting connections
I20260926 12:34:27.781955 10632 mini_ranger.cc:161] Starting Ranger...
I20260926 12:34:27.796756 10632 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 12:34:28.375563 10632 mini_ranger.cc:87] Created ranger Postgres database
I20260926 12:34:28.375653 10632 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ranger-admin
2026-09-26 12:34:28,660 [I] DB FLAVOR :POSTGRES
2026-09-26 12:34:28,660 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 12:34:28,661 [I] Checking connection..
2026-09-26 12:34:28,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 12:34:28,924 [I] Checking connection passed.
2026-09-26 12:34:28,924 [I] --------- Verifying version history table ---------
2026-09-26 12:34:28,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 12:34:29,173 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 12:34:29,173 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 12:34:29,437 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 12:34:29,438 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 12:34:29,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 12:34:29.689 UTC [21505] WARNING: there is no transaction in progress
2026-09-26 12:34:29,695 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 12:34:29,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 12:34:29,955 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 12:34:29,956 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 12:34:29,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 12:34:30,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 12:34:30,453 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-t7tk.c.gcp-upstream.internal','N') ;"
2026-09-26 12:34:30,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 12:34:30,958 [I] Table x_portal_user does not exist in database ranger
2026-09-26 12:34:30,958 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 12:34:31,225 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 12:34:31,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 12:34:31,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 12:34:31,734 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 12:34:31,734 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 12:34:32.784 UTC [21613] WARNING: there is no transaction in progress
2026-09-26 12:34:32.802 UTC [21613] WARNING: there is no transaction in progress
2026-09-26 12:34:32.820 UTC [21613] WARNING: there is no transaction in progress
2026-09-26 12:34:32.836 UTC [21613] WARNING: there is no transaction in progress
2026-09-26 12:34:32.854 UTC [21613] WARNING: there is no transaction in progress
2026-09-26 12:34:32.872 UTC [21613] WARNING: there is no transaction in progress
2026-09-26 12:34:32.932 UTC [21613] WARNING: there is no transaction in progress
2026-09-26 12:34:32.940 UTC [21613] WARNING: there is no transaction in progress
2026-09-26 12:34:32.957 UTC [21613] WARNING: there is no transaction in progress
2026-09-26 12:34:32.969 UTC [21613] WARNING: there is no transaction in progress
2026-09-26 12:34:32.980 UTC [21613] WARNING: there is no transaction in progress
2026-09-26 12:34:32.995 UTC [21613] WARNING: there is no transaction in progress
2026-09-26 12:34:33.007 UTC [21613] WARNING: there is no transaction in progress
2026-09-26 12:34:33.014 UTC [21613] WARNING: there is no transaction in progress
2026-09-26 12:34:33.022 UTC [21613] WARNING: there is no transaction in progress
2026-09-26 12:34:33.030 UTC [21613] WARNING: there is no transaction in progress
2026-09-26 12:34:33,795 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 12:34:33,796 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 12:34:34,049 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 12:34:34,050 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 12:34:34,282 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 12:34:34,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 12:34:34,536 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 12:34:34,536 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 12:34:34,771 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 12:34:35,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-t7tk.c.gcp-upstream.internal';"
2026-09-26 12:34:35,258 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 12:34:35,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 12:34:35,491 [I] DB_PATCHES have already been applied
I20260926 12:34:35.497184 10632 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ranger-admin:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskFMA2CE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 12:34:35.497252 10632 mini_ranger.cc:193] Using host: 127.10.98.20
I20260926 12:34:35.500650 10632 mini_ranger.cc:227] Ranger admin URL: http://127.10.98.20:40127
Sep 26, 2026 12:34:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 12:34:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 12:34:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 12:34:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 12:34:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 12:34:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 12:34:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 12:34:36.510787 10632 mini_ranger.cc:160] Time spent starting Ranger: real 8.729s user 0.000s sys 0.010s
Sep 26, 2026 12:34:57 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 12:34:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 12:34:57 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 12:34:57 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 12:34:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 12:34:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 12:34:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 12:34:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 12:34:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 12:34:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 12:34:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 12:34:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 12:34:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 12:34:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 12:34:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 12:34:57 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 12:34:57 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 12:34:58 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 12:34:58.790552 10632 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/rangerkms_127.10.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/rangerkms_127.10.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/HTTP_127.10.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/HTTP_127.10.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 12:34:58 dist-test-slave-t7tk krb5kdc[21400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426098, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 12:34:58.878552 10632 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 12:34:58.888381 10632 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 12:34:59.882902 10632 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 12:34:59.882997 10632 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ranger-kms
I20260926 12:34:59.883095 10632 mini_ranger_kms.cc:227] Using postgres at 127.10.98.20:41321
2026-09-26 12:35:00,049 [I] DB FLAVOR :POSTGRES
2026-09-26 12:35:00,049 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 12:35:00,049 [I] Checking connection
2026-09-26 12:35:00,049 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 12:35:00,297 [I] connection success
2026-09-26 12:35:00,297 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 12:35:00,297 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 12:35:00,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 12:35:00,560 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 12:35:00,561 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 12:35:00,561 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 12:35:00,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:41321/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 12:35:00,821 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 12:35:01.271287 10632 mini_ranger_kms.cc:310] Created kms service
I20260926 12:35:01.445662 10632 mini_ranger_kms.cc:326] Created kudu user
I20260926 12:35:01.490198 10632 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 12:35:01 dist-test-slave-t7tk krb5kdc[21400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426098, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
I20260926 12:35:01.883569 10632 mini_ranger_kms.cc:384] Added ranger policy
I20260926 12:35:01.883729 10632 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ranger-kms:/tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 12:35:01.883754 10632 mini_ranger_kms.cc:240] Using host: 127.10.98.20
I20260926 12:35:01.887147 10632 mini_ranger_kms.cc:276] Ranger KMS PID: 21793
I20260926 12:35:01.887286 10632 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.98.20:47991
Sep 26, 2026 12:35:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 12:35:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 12:35:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 12:35:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 12:35:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 12:35:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 12:35:03.912737 10632 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.034s user 0.006s sys 0.143s
I20260926 12:35:03.912859 10632 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 12:35:03.912974 10632 mini_ranger_kms.cc:401] 127.10.98.20:47991/kms/v1/keys
Sep 26 12:35:04 dist-test-slave-t7tk krb5kdc[21400](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790426104, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 12:35:04 dist-test-slave-t7tk krb5kdc[21400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426104, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.98.20@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
Sep 26 12:35:04 dist-test-slave-t7tk krb5kdc[21400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426104, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.98.20@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
Sep 26 12:35:07 dist-test-slave-t7tk krb5kdc[21400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:35:07.107890 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:44839
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:45975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:44839
--ranger_config_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.98.20:47991/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:35:07.228672 21844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:35:07.228976 21844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:35:07.229034 21844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:35:07.233237 21844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:35:07.233310 21844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:35:07.233335 21844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:35:07.233358 21844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:35:07.233386 21844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:35:07.239073 21844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:45975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.98.20:47991/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:44839
--ranger_config_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:44839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:35:07.240284 21844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:35:07.241441 21844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:35:07.250377 21849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:35:07.250684 21844 server_base.cc:1034] running on GCE node
W20260926 12:35:07.250707 21852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:35:07.250738 21850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:35:07.251430 21844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:35:07.252595 21844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:35:07.253749 21844 hybrid_clock.cc:648] HybridClock initialized: now 1790426107253733 us; error 46 us; skew 500 ppm
Sep 26 12:35:07 dist-test-slave-t7tk krb5kdc[21400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426107, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:35:07.256537 21844 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:35:07.257550 21844 webserver.cc:460] Webserver started at http://127.10.98.62:44571/ using document root <none> and password file <none>
I20260926 12:35:07.258100 21844 fs_manager.cc:359] Metadata directory not provided
I20260926 12:35:07.258155 21844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:35:07.258355 21844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:35:07 dist-test-slave-t7tk krb5kdc[21400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426107, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
I20260926 12:35:07.306813 21844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "3e77ac6652b94caba18b1ae2afdd28ef"
format_stamp: "Formatted at 2026-09-26 12:35:07 on dist-test-slave-t7tk"
server_key: "da9802d41ada5d03d3cf1d8f71943505"
server_key_iv: "0bde9124d67f4eda65f62b3fafda6a37"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:07.307482 21844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "3e77ac6652b94caba18b1ae2afdd28ef"
format_stamp: "Formatted at 2026-09-26 12:35:07 on dist-test-slave-t7tk"
server_key: "da9802d41ada5d03d3cf1d8f71943505"
server_key_iv: "0bde9124d67f4eda65f62b3fafda6a37"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:07.311749 21844 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.003s
I20260926 12:35:07.326508 21859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:07.327951 21844 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:35:07.328114 21844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "3e77ac6652b94caba18b1ae2afdd28ef"
format_stamp: "Formatted at 2026-09-26 12:35:07 on dist-test-slave-t7tk"
server_key: "da9802d41ada5d03d3cf1d8f71943505"
server_key_iv: "0bde9124d67f4eda65f62b3fafda6a37"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:07.328244 21844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:35:07.353695 21844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:35:07.356851 21844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:35:07.357038 21844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:35:07.364738 21844 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:44839
I20260926 12:35:07.364787 21911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:44839 every 8 connection(s)
I20260926 12:35:07.365832 21844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:35:07.368827 21912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:35:07.374742 21912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef: Bootstrap starting.
I20260926 12:35:07.375370 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 21844
I20260926 12:35:07.375501 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:35:07.375792 10632 external_mini_cluster.cc:1428] Setting key f0b228fe30f07729f9e537a55bbe1f2f
I20260926 12:35:07.377882 21912 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef: Neither blocks nor log segments found. Creating new log.
I20260926 12:35:07.378852 21912 log.cc:824] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef: Log is configured to *not* fsync() on all Append() calls
I20260926 12:35:07.381220 21912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef: No bootstrap required, opened a new log
Sep 26 12:35:07 dist-test-slave-t7tk krb5kdc[21400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:35:07.386212 21912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e77ac6652b94caba18b1ae2afdd28ef" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 44839 } }
I20260926 12:35:07.386523 21912 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:35:07.386581 21912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e77ac6652b94caba18b1ae2afdd28ef, State: Initialized, Role: FOLLOWER
I20260926 12:35:07.387169 21912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e77ac6652b94caba18b1ae2afdd28ef" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 44839 } }
I20260926 12:35:07.387344 21912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:35:07.387434 21912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:35:07.387538 21912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:35:07.388684 21912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e77ac6652b94caba18b1ae2afdd28ef" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 44839 } }
I20260926 12:35:07.389173 21912 leader_election.cc:302] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e77ac6652b94caba18b1ae2afdd28ef; no voters:
I20260926 12:35:07.389595 21912 leader_election.cc:288] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:35:07.389700 21917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:35:07.389967 21917 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef [term 1 LEADER]: Becoming Leader. State: Replica: 3e77ac6652b94caba18b1ae2afdd28ef, State: Running, Role: LEADER
I20260926 12:35:07.390357 21917 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e77ac6652b94caba18b1ae2afdd28ef" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 44839 } }
I20260926 12:35:07.390555 21915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:07.377294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.10.98.62:44839)
0926 12:35:07.377793 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:07.377813 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:07.377860 (+ 47us) server_negotiation.cc:407] Connection header received
0926 12:35:07.378740 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:07.378764 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:07.379223 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:07.379584 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:35:07.380407 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:07.381566 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:07.382484 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:07.382975 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:07.383120 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:07.385579 (+ 2459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:35:07.385585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:35:07.385600 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:35:07.385629 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:35:07.387284 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:07.387829 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:07.387835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:07.387841 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:07.387958 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:07.388219 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:07.388222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:07.388224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:07.388650 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:35:07.388913 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:07.389149 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 12:35:07.389501 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":152,"threads_started":1}
I20260926 12:35:07.391906 21917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e77ac6652b94caba18b1ae2afdd28ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e77ac6652b94caba18b1ae2afdd28ef" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 44839 } } }
I20260926 12:35:07.392103 21912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:35:07.392114 21917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef [sys.catalog]: This master's current role is: LEADER
I20260926 12:35:07.392334 21917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e77ac6652b94caba18b1ae2afdd28ef. Latest consensus state: current_term: 1 leader_uuid: "3e77ac6652b94caba18b1ae2afdd28ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e77ac6652b94caba18b1ae2afdd28ef" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 44839 } } }
I20260926 12:35:07.392400 21917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef [sys.catalog]: This master's current role is: LEADER
I20260926 12:35:07.392999 21921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:35:07.396662 21921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:35:07.397152 21912 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 12:35:07.411191 21921 catalog_manager.cc:1347] Generated new cluster ID: 38252ee1853449f4aed7f5e8183f37d8
I20260926 12:35:07.411293 21921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:35:07.436805 21921 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:35:07.438313 21921 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:35:07.454715 21921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef: Generated new TSK 0
I20260926 12:35:07.455650 21921 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
I20260926 12:35:08.100009 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.98.20:47991/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:44839
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:45975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:35:08.238711 21957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:35:08.238986 21957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:35:08.239045 21957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:35:08.243261 21957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:35:08.243331 21957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:35:08.243413 21957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:35:08.248976 21957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:45975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.98.20:47991/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:44839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:35:08.250329 21957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:35:08.251708 21957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:35:08.261291 21965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:35:08.261516 21957 server_base.cc:1034] running on GCE node
W20260926 12:35:08.261718 21967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:35:08.261719 21964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:35:08.262210 21957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:35:08.262842 21957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:35:08.264057 21957 hybrid_clock.cc:648] HybridClock initialized: now 1790426108264023 us; error 43 us; skew 500 ppm
Sep 26 12:35:08 dist-test-slave-t7tk krb5kdc[21400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426108, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:35:08.266758 21957 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:35:08.267637 21957 webserver.cc:460] Webserver started at http://127.10.98.1:40827/ using document root <none> and password file <none>
I20260926 12:35:08.268262 21957 fs_manager.cc:359] Metadata directory not provided
I20260926 12:35:08.268330 21957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:35:08.268529 21957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:35:08 dist-test-slave-t7tk krb5kdc[21400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426108, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
I20260926 12:35:08.284823 21957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "ea3ab5c1b9d9462391bbe5c9c17bf893"
format_stamp: "Formatted at 2026-09-26 12:35:08 on dist-test-slave-t7tk"
server_key: "c04824fac4b4190294a49913804baf86"
server_key_iv: "325d24e84c5317535e6ec01b8a82c0ae"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:08.285856 21957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "ea3ab5c1b9d9462391bbe5c9c17bf893"
format_stamp: "Formatted at 2026-09-26 12:35:08 on dist-test-slave-t7tk"
server_key: "c04824fac4b4190294a49913804baf86"
server_key_iv: "325d24e84c5317535e6ec01b8a82c0ae"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:08.290407 21957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:35:08.313540 21974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:08.315397 21957 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260926 12:35:08.315563 21957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "ea3ab5c1b9d9462391bbe5c9c17bf893"
format_stamp: "Formatted at 2026-09-26 12:35:08 on dist-test-slave-t7tk"
server_key: "c04824fac4b4190294a49913804baf86"
server_key_iv: "325d24e84c5317535e6ec01b8a82c0ae"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:08.315709 21957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:35:08.347491 21957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:35:08.351472 21957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:35:08.351727 21957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:35:08.352536 21957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:35:08.353655 21957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:35:08.353757 21957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:08.353889 21957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:35:08.353978 21957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:08.367772 21957 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:35023
I20260926 12:35:08.369594 21957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:35:08.371004 22089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:35023 every 8 connection(s)
I20260926 12:35:08.379426 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 21957
I20260926 12:35:08.379557 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:35:08.379843 10632 external_mini_cluster.cc:1428] Setting key ea620ed0ee9e3328be8eb339aa6185ac
Sep 26 12:35:08 dist-test-slave-t7tk krb5kdc[21400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426108, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:35:08.392901 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:08.374444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:41857 (local address 127.10.98.62:44839)
0926 12:35:08.376933 (+ 2489us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:08.376942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:08.376957 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:35:08.376999 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:08.377002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:08.377050 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:08.377450 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:35:08.379035 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:08.380935 (+ 1900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:08.384675 (+ 3740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:08.385033 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:08.385130 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:08.388495 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:35:08.388499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:35:08.388502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:35:08.388530 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:35:08.390329 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:08.390975 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:08.390979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:08.390981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:08.391072 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:08.391491 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:08.391494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:08.391496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:08.391747 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:35:08.391823 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:08.392576 (+ 753us) server_negotiation.cc:299] Negotiation successful
0926 12:35:08.392713 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2409,"thread_start_us":109,"threads_started":1}
I20260926 12:35:08.394964 22092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:08.373347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:44839 (local address 127.10.98.1:41857)
0926 12:35:08.375315 (+ 1968us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:08.375437 (+ 122us) client_negotiation.cc:174] Beginning negotiation
0926 12:35:08.376488 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:35:08.377792 (+ 1304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:35:08.377801 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:35:08.378170 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:35:08.378848 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:08.378882 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:08.381071 (+ 2189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:08.381074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:08.384524 (+ 3450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:08.384537 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:08.385342 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:08.385345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:08.385671 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:08.386471 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:35:08.386499 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:35:08.388257 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:35:08.390491 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:35:08.390498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:35:08.390516 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:35:08.390856 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:35:08.391173 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:35:08.391176 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:35:08.391178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:35:08.391369 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:35:08.391907 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:35:08.391914 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:35:08.392267 (+ 353us) client_negotiation.cc:769] Sending connection context
0926 12:35:08.393509 (+ 1242us) client_negotiation.cc:240] Negotiation successful
0926 12:35:08.393877 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1683,"thread_start_us":126,"threads_started":1}
I20260926 12:35:08.398069 22090 heartbeater.cc:347] Connected to a master server at 127.10.98.62:44839
I20260926 12:35:08.398344 22090 heartbeater.cc:464] Registering TS with master...
I20260926 12:35:08.398969 22090 heartbeater.cc:511] Master 127.10.98.62:44839 requested a full tablet report, sending...
I20260926 12:35:08.400851 21876 ts_manager.cc:194] Registered new tserver with Master: ea3ab5c1b9d9462391bbe5c9c17bf893 (127.10.98.1:35023)
I20260926 12:35:08.402580 21876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:41857
WARNING: no policy specified for kudu/127.10.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
I20260926 12:35:08.433022 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.98.20:47991/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:44839
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:45975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:35:08 dist-test-slave-t7tk krb5kdc[21400](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790426108, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 12:35:08.602885 22096 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:35:08.603175 22096 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:35:08.603235 22096 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:35:08.607678 22096 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:35:08.607743 22096 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:35:08.607823 22096 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:35:08.615962 22096 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:45975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.98.20:47991/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:44839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22096
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:35:08.617585 22096 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:35:08.619001 22096 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:35:08.635674 22105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:35:08.637594 22103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:35:08.637843 22102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:35:08.638701 22096 server_base.cc:1034] running on GCE node
I20260926 12:35:08.639238 22096 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:35:08.640146 22096 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:35:08.644701 22096 hybrid_clock.cc:648] HybridClock initialized: now 1790426108644599 us; error 99 us; skew 500 ppm
Sep 26 12:35:08 dist-test-slave-t7tk krb5kdc[21400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426108, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:35:08.649055 22096 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:35:08.650470 22096 webserver.cc:460] Webserver started at http://127.10.98.2:41367/ using document root <none> and password file <none>
I20260926 12:35:08.651458 22096 fs_manager.cc:359] Metadata directory not provided
I20260926 12:35:08.654765 22096 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:35:08.655186 22096 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:35:08 dist-test-slave-t7tk krb5kdc[21400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426108, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
I20260926 12:35:08.681630 22096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "08438c15d8e6447d8522d3dfe57b2fa5"
format_stamp: "Formatted at 2026-09-26 12:35:08 on dist-test-slave-t7tk"
server_key: "9cf54224b75d61cb2ede9b5ac65d0649"
server_key_iv: "0b358a00d08acb015b54e29572133a8e"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:08.684630 22096 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "08438c15d8e6447d8522d3dfe57b2fa5"
format_stamp: "Formatted at 2026-09-26 12:35:08 on dist-test-slave-t7tk"
server_key: "9cf54224b75d61cb2ede9b5ac65d0649"
server_key_iv: "0b358a00d08acb015b54e29572133a8e"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:08.690685 22096 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260926 12:35:08.716147 22112 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:08.718055 22096 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260926 12:35:08.718201 22096 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "08438c15d8e6447d8522d3dfe57b2fa5"
format_stamp: "Formatted at 2026-09-26 12:35:08 on dist-test-slave-t7tk"
server_key: "9cf54224b75d61cb2ede9b5ac65d0649"
server_key_iv: "0b358a00d08acb015b54e29572133a8e"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:08.718340 22096 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:35:08.733371 22096 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:35:08.736783 22096 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:35:08.736963 22096 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:35:08.737599 22096 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:35:08.738487 22096 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:35:08.738536 22096 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:08.738587 22096 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:35:08.738617 22096 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:08.749008 22096 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:37777
I20260926 12:35:08.750684 22096 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:35:08.754352 22225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:37777 every 8 connection(s)
I20260926 12:35:08.760703 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 22096
I20260926 12:35:08.760859 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:35:08.761175 10632 external_mini_cluster.cc:1428] Setting key b6df680e9d774be104f4b170ec772c63
Sep 26 12:35:08 dist-test-slave-t7tk krb5kdc[21400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426108, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:35:08.779572 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:08.759309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:34407 (local address 127.10.98.62:44839)
0926 12:35:08.759452 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:08.759457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:08.765648 (+ 6191us) server_negotiation.cc:407] Connection header received
0926 12:35:08.767059 (+ 1411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:08.767063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:08.767125 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:08.767182 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:35:08.768955 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:08.769832 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:08.771591 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:08.771958 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:08.772040 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:08.775476 (+ 3436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:35:08.775480 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:35:08.775482 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:35:08.775516 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:35:08.777154 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:08.777795 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:08.777798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:08.777800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:08.777884 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:08.778279 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:08.778282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:08.778284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:08.778503 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:35:08.778569 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:08.779265 (+ 696us) server_negotiation.cc:299] Negotiation successful
0926 12:35:08.779401 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:35:08.781219 22228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:08.759824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:44839 (local address 127.10.98.2:34407)
0926 12:35:08.765468 (+ 5644us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:08.765528 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 12:35:08.766731 (+ 1203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:35:08.767636 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:35:08.767646 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:35:08.768048 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:35:08.768761 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:08.768782 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:08.769996 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:08.769999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:08.771445 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:08.771457 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:08.772257 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:08.772260 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:08.772469 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:08.773454 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:35:08.773476 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:35:08.775255 (+ 1779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:35:08.777298 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:35:08.777304 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:35:08.777323 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:35:08.777677 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:35:08.777998 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:35:08.778000 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:35:08.778002 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:35:08.778179 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:35:08.778644 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:35:08.778650 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:35:08.778998 (+ 348us) client_negotiation.cc:769] Sending connection context
0926 12:35:08.779745 (+ 747us) client_negotiation.cc:240] Negotiation successful
0926 12:35:08.780116 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5384,"thread_start_us":152,"threads_started":1}
I20260926 12:35:08.782712 22226 heartbeater.cc:347] Connected to a master server at 127.10.98.62:44839
I20260926 12:35:08.782998 22226 heartbeater.cc:464] Registering TS with master...
I20260926 12:35:08.783598 22226 heartbeater.cc:511] Master 127.10.98.62:44839 requested a full tablet report, sending...
I20260926 12:35:08.784842 21876 ts_manager.cc:194] Registered new tserver with Master: 08438c15d8e6447d8522d3dfe57b2fa5 (127.10.98.2:37777)
WARNING: no policy specified for kudu/127.10.98.3@KRBTEST.COM; defaulting to no policy
I20260926 12:35:08.785600 21876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:34407
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
I20260926 12:35:08.814425 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.98.20:47991/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:44839
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:45975
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:35:08.996560 22231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:35:08.997014 22231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:35:08.997157 22231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:35:09.003925 22231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:35:09.004096 22231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:35:09.004257 22231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
Sep 26 12:35:09 dist-test-slave-t7tk krb5kdc[21400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426108, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
I20260926 12:35:09.012259 22231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:45975
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.98.20:47991/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:44839
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:35:09.014081 22231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:35:09.015434 22231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:35:09.029315 22237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:35:09.029320 22239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:35:09.030900 22236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:35:09.038527 22231 server_base.cc:1034] running on GCE node
I20260926 12:35:09.039240 22231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:35:09.040102 22231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:35:09.055593 22231 hybrid_clock.cc:648] HybridClock initialized: now 1790426109055486 us; error 117 us; skew 500 ppm
Sep 26 12:35:09 dist-test-slave-t7tk krb5kdc[21400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426109, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:35:09.059800 22231 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:35:09.060984 22231 webserver.cc:460] Webserver started at http://127.10.98.3:34913/ using document root <none> and password file <none>
I20260926 12:35:09.061754 22231 fs_manager.cc:359] Metadata directory not provided
I20260926 12:35:09.061839 22231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:35:09.062099 22231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:35:09 dist-test-slave-t7tk krb5kdc[21400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426109, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
I20260926 12:35:09.088073 22231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "ec8c24d5e388486eacb880ecce669007"
format_stamp: "Formatted at 2026-09-26 12:35:09 on dist-test-slave-t7tk"
server_key: "a4e666c7549fdf03f92a353ca443d6e4"
server_key_iv: "3006d034bc6974e57b806aae846cc78b"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:09.092893 22231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "ec8c24d5e388486eacb880ecce669007"
format_stamp: "Formatted at 2026-09-26 12:35:09 on dist-test-slave-t7tk"
server_key: "a4e666c7549fdf03f92a353ca443d6e4"
server_key_iv: "3006d034bc6974e57b806aae846cc78b"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:09.114822 22231 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.001s sys 0.003s
I20260926 12:35:09.146519 22246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:09.150804 22231 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.003s sys 0.001s
I20260926 12:35:09.150950 22231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "ec8c24d5e388486eacb880ecce669007"
format_stamp: "Formatted at 2026-09-26 12:35:09 on dist-test-slave-t7tk"
server_key: "a4e666c7549fdf03f92a353ca443d6e4"
server_key_iv: "3006d034bc6974e57b806aae846cc78b"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:09.151093 22231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:35:09.171700 22231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:35:09.180019 22231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:35:09.184685 22231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:35:09.185545 22231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:35:09.186636 22231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:35:09.186739 22231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:09.186830 22231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:35:09.186969 22231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:09.196700 22231 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:35827
I20260926 12:35:09.202071 22231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:35:09.200773 22359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:35827 every 8 connection(s)
I20260926 12:35:09.207398 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 22231
I20260926 12:35:09.207515 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:35:09.207787 10632 external_mini_cluster.cc:1428] Setting key 8ecc4ced7eb5f529d3001f168e69fcce
Sep 26 12:35:09 dist-test-slave-t7tk krb5kdc[21400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426109, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:35:09.225016 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:09.206670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:56961 (local address 127.10.98.62:44839)
0926 12:35:09.206795 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:09.206800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:09.210076 (+ 3276us) server_negotiation.cc:407] Connection header received
0926 12:35:09.211370 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:09.211373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:09.211430 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:09.211490 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:35:09.213178 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.213971 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.215631 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.215977 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.216047 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:09.219001 (+ 2954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:35:09.219005 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:35:09.219008 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:35:09.219036 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:35:09.222450 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:09.223197 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:09.223200 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:09.223202 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:09.223285 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:09.223658 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:09.223661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:09.223663 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:09.223900 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:35:09.223987 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:09.224714 (+ 727us) server_negotiation.cc:299] Negotiation successful
0926 12:35:09.224840 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:35:09.226915 22362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:09.205479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:44839 (local address 127.10.98.3:56961)
0926 12:35:09.207116 (+ 1637us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:09.208238 (+ 1122us) client_negotiation.cc:174] Beginning negotiation
0926 12:35:09.211088 (+ 2850us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:35:09.211930 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:35:09.211939 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:35:09.212305 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:35:09.212993 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:09.213012 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.214123 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.214126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:09.215500 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:09.215511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.216246 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.216250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:09.216444 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:09.217138 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:35:09.217159 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:35:09.218797 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:35:09.222654 (+ 3857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:35:09.222660 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:35:09.222680 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:35:09.223089 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:35:09.223377 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:35:09.223380 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:35:09.223382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:35:09.223559 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:35:09.224103 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:35:09.224110 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:35:09.224443 (+ 333us) client_negotiation.cc:769] Sending connection context
0926 12:35:09.225507 (+ 1064us) client_negotiation.cc:240] Negotiation successful
0926 12:35:09.225856 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1393,"thread_start_us":161,"threads_started":1}
I20260926 12:35:09.228458 22360 heartbeater.cc:347] Connected to a master server at 127.10.98.62:44839
I20260926 12:35:09.228705 22360 heartbeater.cc:464] Registering TS with master...
I20260926 12:35:09.229297 22360 heartbeater.cc:511] Master 127.10.98.62:44839 requested a full tablet report, sending...
I20260926 12:35:09.230916 21876 ts_manager.cc:194] Registered new tserver with Master: ec8c24d5e388486eacb880ecce669007 (127.10.98.3:35827)
I20260926 12:35:09.231696 21876 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:56961
I20260926 12:35:09.233748 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:35:09.261396 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:09.236441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45512 (local address 127.10.98.62:44839)
0926 12:35:09.236574 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:09.236580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:09.237405 (+ 825us) server_negotiation.cc:407] Connection header received
0926 12:35:09.237617 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:09.237621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:09.237672 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:09.237723 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:35:09.238666 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.239446 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.240703 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.241027 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.241143 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:09.257629 (+ 16486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:35:09.257634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:35:09.257637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:35:09.257674 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:35:09.259323 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:09.260061 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:09.260065 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:09.260067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:09.260150 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:09.260427 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:09.260430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:09.260432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:09.260644 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:35:09.260737 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:09.261093 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 12:35:09.261233 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:35:09.265242 21876 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45512:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:35:09.269506 21876 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:35:09.326815 22375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:09.316897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:35827 (local address 127.0.0.1:50720)
0926 12:35:09.319419 (+ 2522us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:09.319429 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 12:35:09.319490 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:35:09.320186 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:35:09.320191 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:35:09.320213 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:35:09.321043 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:09.321051 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.322534 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.322537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:09.325008 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:09.325018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.326309 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.326312 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:09.326410 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:09.326449 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 12:35:09.326550 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 12:35:09.326666 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2478,"thread_start_us":84,"threads_started":1}
I20260926 12:35:09.328656 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:09.317292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50720 (local address 127.10.98.3:35827)
0926 12:35:09.319078 (+ 1786us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:09.319089 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:09.319572 (+ 483us) server_negotiation.cc:407] Connection header received
0926 12:35:09.319652 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:09.319657 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:09.319864 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:09.320017 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:35:09.321222 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.322425 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.325171 (+ 2746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.326010 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.326110 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:09.326253 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:09.328376 (+ 2123us) server_negotiation.cc:299] Negotiation successful
0926 12:35:09.328498 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1686,"thread_start_us":99,"threads_started":1}
I20260926 12:35:09.328644 22377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:09.314463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:37777 (local address 127.0.0.1:48066)
0926 12:35:09.318629 (+ 4166us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:09.318682 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 12:35:09.318804 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:35:09.321897 (+ 3093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:35:09.321901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:35:09.321916 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:35:09.322026 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:09.322033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.325227 (+ 3194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.325230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:09.326294 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:09.326301 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.328258 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.328261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:09.328352 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:09.328374 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:35:09.328436 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 12:35:09.328513 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3986,"thread_start_us":99,"threads_started":1}
I20260926 12:35:09.332144 22294 tablet_service.cc:1467] Processing CreateTablet for tablet 85fbc31ba61d41a7a889e25a895cc11e (DEFAULT_TABLE table=test-table [id=7fbd2ec0979f4fc485fd528da5ce9d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:35:09.334422 22294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85fbc31ba61d41a7a889e25a895cc11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:35:09.338881 22374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:09.314583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48066 (local address 127.10.98.2:37777)
0926 12:35:09.321300 (+ 6717us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:09.321307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:09.321331 (+ 24us) server_negotiation.cc:407] Connection header received
0926 12:35:09.321401 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:09.321405 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:09.321615 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:09.321761 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:35:09.323932 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.325117 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.327162 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.327956 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.328054 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:09.328200 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:09.338585 (+ 10385us) server_negotiation.cc:299] Negotiation successful
0926 12:35:09.338723 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6600,"thread_start_us":73,"threads_started":1}
I20260926 12:35:09.342131 22142 tablet_service.cc:1467] Processing CreateTablet for tablet 85fbc31ba61d41a7a889e25a895cc11e (DEFAULT_TABLE table=test-table [id=7fbd2ec0979f4fc485fd528da5ce9d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:35:09.343210 22373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:09.316322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:35023 (local address 127.0.0.1:36678)
0926 12:35:09.319202 (+ 2880us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:09.319213 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:35:09.319284 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:35:09.327169 (+ 7885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:35:09.327172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:35:09.327889 (+ 717us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:35:09.327990 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:09.327997 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.340071 (+ 12074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.340074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:09.341316 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:09.341329 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.342771 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.342774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:09.342877 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:09.342902 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 12:35:09.342984 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 12:35:09.343067 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2827,"thread_start_us":99,"threads_started":1}
I20260926 12:35:09.343264 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:09.316510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36678 (local address 127.10.98.1:35023)
0926 12:35:09.326566 (+ 10056us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:09.326573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:09.326596 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:35:09.326666 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:09.326671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:09.326873 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:09.327020 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:35:09.337191 (+ 10171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.338364 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.341442 (+ 3078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.342269 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.342379 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:09.342543 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:09.343036 (+ 493us) server_negotiation.cc:299] Negotiation successful
0926 12:35:09.343140 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9929,"thread_start_us":97,"threads_started":1}
I20260926 12:35:09.343278 22142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85fbc31ba61d41a7a889e25a895cc11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:35:09.346481 22024 tablet_service.cc:1467] Processing CreateTablet for tablet 85fbc31ba61d41a7a889e25a895cc11e (DEFAULT_TABLE table=test-table [id=7fbd2ec0979f4fc485fd528da5ce9d8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:35:09.347613 22024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85fbc31ba61d41a7a889e25a895cc11e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:35:09.357724 22381 tablet_bootstrap.cc:492] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007: Bootstrap starting.
I20260926 12:35:09.361598 22379 tablet_bootstrap.cc:492] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5: Bootstrap starting.
I20260926 12:35:09.364755 22379 tablet_bootstrap.cc:654] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5: Neither blocks nor log segments found. Creating new log.
I20260926 12:35:09.365293 22380 tablet_bootstrap.cc:492] T 85fbc31ba61d41a7a889e25a895cc11e P ea3ab5c1b9d9462391bbe5c9c17bf893: Bootstrap starting.
I20260926 12:35:09.365777 22379 log.cc:824] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5: Log is configured to *not* fsync() on all Append() calls
I20260926 12:35:09.367902 22380 tablet_bootstrap.cc:654] T 85fbc31ba61d41a7a889e25a895cc11e P ea3ab5c1b9d9462391bbe5c9c17bf893: Neither blocks nor log segments found. Creating new log.
I20260926 12:35:09.379415 22379 tablet_bootstrap.cc:492] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5: No bootstrap required, opened a new log
I20260926 12:35:09.379673 22379 ts_tablet_manager.cc:1397] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260926 12:35:09.380774 22381 tablet_bootstrap.cc:654] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007: Neither blocks nor log segments found. Creating new log.
I20260926 12:35:09.381175 22380 log.cc:824] T 85fbc31ba61d41a7a889e25a895cc11e P ea3ab5c1b9d9462391bbe5c9c17bf893: Log is configured to *not* fsync() on all Append() calls
I20260926 12:35:09.381745 22381 log.cc:824] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007: Log is configured to *not* fsync() on all Append() calls
I20260926 12:35:09.384073 22379 raft_consensus.cc:348] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08438c15d8e6447d8522d3dfe57b2fa5" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37777 } } peers { permanent_uuid: "ea3ab5c1b9d9462391bbe5c9c17bf893" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35023 } } peers { permanent_uuid: "ec8c24d5e388486eacb880ecce669007" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 35827 } }
I20260926 12:35:09.384364 22379 raft_consensus.cc:374] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:35:09.384414 22379 raft_consensus.cc:729] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08438c15d8e6447d8522d3dfe57b2fa5, State: Initialized, Role: FOLLOWER
I20260926 12:35:09.384987 22379 consensus_queue.cc:260] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08438c15d8e6447d8522d3dfe57b2fa5" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37777 } } peers { permanent_uuid: "ea3ab5c1b9d9462391bbe5c9c17bf893" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35023 } } peers { permanent_uuid: "ec8c24d5e388486eacb880ecce669007" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 35827 } }
I20260926 12:35:09.386063 22379 ts_tablet_manager.cc:1428] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:35:09.391268 22380 tablet_bootstrap.cc:492] T 85fbc31ba61d41a7a889e25a895cc11e P ea3ab5c1b9d9462391bbe5c9c17bf893: No bootstrap required, opened a new log
I20260926 12:35:09.391570 22380 ts_tablet_manager.cc:1397] T 85fbc31ba61d41a7a889e25a895cc11e P ea3ab5c1b9d9462391bbe5c9c17bf893: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.000s
I20260926 12:35:09.393303 22226 heartbeater.cc:503] Master 127.10.98.62:44839 was elected leader, sending a full tablet report...
I20260926 12:35:09.394268 22381 tablet_bootstrap.cc:492] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007: No bootstrap required, opened a new log
I20260926 12:35:09.394483 22381 ts_tablet_manager.cc:1397] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007: Time spent bootstrapping tablet: real 0.037s user 0.007s sys 0.000s
I20260926 12:35:09.395690 22380 raft_consensus.cc:348] T 85fbc31ba61d41a7a889e25a895cc11e P ea3ab5c1b9d9462391bbe5c9c17bf893 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08438c15d8e6447d8522d3dfe57b2fa5" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37777 } } peers { permanent_uuid: "ea3ab5c1b9d9462391bbe5c9c17bf893" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35023 } } peers { permanent_uuid: "ec8c24d5e388486eacb880ecce669007" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 35827 } }
I20260926 12:35:09.396068 22380 raft_consensus.cc:374] T 85fbc31ba61d41a7a889e25a895cc11e P ea3ab5c1b9d9462391bbe5c9c17bf893 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:35:09.396162 22380 raft_consensus.cc:729] T 85fbc31ba61d41a7a889e25a895cc11e P ea3ab5c1b9d9462391bbe5c9c17bf893 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea3ab5c1b9d9462391bbe5c9c17bf893, State: Initialized, Role: FOLLOWER
I20260926 12:35:09.396734 22380 consensus_queue.cc:260] T 85fbc31ba61d41a7a889e25a895cc11e P ea3ab5c1b9d9462391bbe5c9c17bf893 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08438c15d8e6447d8522d3dfe57b2fa5" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37777 } } peers { permanent_uuid: "ea3ab5c1b9d9462391bbe5c9c17bf893" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35023 } } peers { permanent_uuid: "ec8c24d5e388486eacb880ecce669007" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 35827 } }
I20260926 12:35:09.397969 22380 ts_tablet_manager.cc:1428] T 85fbc31ba61d41a7a889e25a895cc11e P ea3ab5c1b9d9462391bbe5c9c17bf893: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:35:09.399715 22381 raft_consensus.cc:348] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08438c15d8e6447d8522d3dfe57b2fa5" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37777 } } peers { permanent_uuid: "ea3ab5c1b9d9462391bbe5c9c17bf893" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35023 } } peers { permanent_uuid: "ec8c24d5e388486eacb880ecce669007" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 35827 } }
I20260926 12:35:09.399994 22381 raft_consensus.cc:374] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:35:09.400049 22381 raft_consensus.cc:729] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec8c24d5e388486eacb880ecce669007, State: Initialized, Role: FOLLOWER
I20260926 12:35:09.400564 22381 consensus_queue.cc:260] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08438c15d8e6447d8522d3dfe57b2fa5" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37777 } } peers { permanent_uuid: "ea3ab5c1b9d9462391bbe5c9c17bf893" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35023 } } peers { permanent_uuid: "ec8c24d5e388486eacb880ecce669007" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 35827 } }
I20260926 12:35:09.401404 22090 heartbeater.cc:503] Master 127.10.98.62:44839 was elected leader, sending a full tablet report...
I20260926 12:35:09.401587 22381 ts_tablet_manager.cc:1428] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 12:35:09.401909 22360 heartbeater.cc:503] Master 127.10.98.62:44839 was elected leader, sending a full tablet report...
W20260926 12:35:09.453570 22361 tablet.cc:2369] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:35:09.508318 22227 tablet.cc:2369] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:35:09.532932 22387 raft_consensus.cc:482] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:35:09.533182 22387 raft_consensus.cc:504] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08438c15d8e6447d8522d3dfe57b2fa5" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37777 } } peers { permanent_uuid: "ea3ab5c1b9d9462391bbe5c9c17bf893" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35023 } } peers { permanent_uuid: "ec8c24d5e388486eacb880ecce669007" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 35827 } }
I20260926 12:35:09.534538 22387 leader_election.cc:288] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 08438c15d8e6447d8522d3dfe57b2fa5 (127.10.98.2:37777), ea3ab5c1b9d9462391bbe5c9c17bf893 (127.10.98.1:35023)
I20260926 12:35:09.540949 22362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:09.534835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:37777 (local address 127.10.98.3:47543)
0926 12:35:09.535450 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:09.535466 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:35:09.535588 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:35:09.537029 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:35:09.537033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:35:09.537054 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:35:09.537198 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:09.537205 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.538215 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.538218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:09.539503 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:09.539515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.540516 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.540520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:09.540623 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:09.540648 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 12:35:09.540720 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 12:35:09.540800 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520}
I20260926 12:35:09.541503 22374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:09.535961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:47543 (local address 127.10.98.2:37777)
0926 12:35:09.536217 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:09.536222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:09.536237 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:35:09.536295 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:09.536310 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:09.536362 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:09.536425 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:35:09.537345 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.538081 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.539649 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.540269 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.540384 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:09.540455 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:09.541289 (+ 834us) server_negotiation.cc:299] Negotiation successful
0926 12:35:09.541383 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260926 12:35:09.542197 22162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85fbc31ba61d41a7a889e25a895cc11e" candidate_uuid: "ec8c24d5e388486eacb880ecce669007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08438c15d8e6447d8522d3dfe57b2fa5" is_pre_election: true
Sep 26 12:35:09 dist-test-slave-t7tk krb5kdc[21400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426108, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
I20260926 12:35:09.542564 22162 raft_consensus.cc:2454] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8c24d5e388486eacb880ecce669007 in term 0.
I20260926 12:35:09.543473 22248 leader_election.cc:302] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08438c15d8e6447d8522d3dfe57b2fa5, ec8c24d5e388486eacb880ecce669007; no voters:
I20260926 12:35:09.543803 22387 raft_consensus.cc:2781] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:35:09.543881 22387 raft_consensus.cc:482] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:35:09.543919 22387 raft_consensus.cc:3037] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:35:09.545471 22387 raft_consensus.cc:504] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08438c15d8e6447d8522d3dfe57b2fa5" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37777 } } peers { permanent_uuid: "ea3ab5c1b9d9462391bbe5c9c17bf893" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35023 } } peers { permanent_uuid: "ec8c24d5e388486eacb880ecce669007" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 35827 } }
I20260926 12:35:09.545886 22387 leader_election.cc:288] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 [CANDIDATE]: Term 1 election: Requested vote from peers 08438c15d8e6447d8522d3dfe57b2fa5 (127.10.98.2:37777), ea3ab5c1b9d9462391bbe5c9c17bf893 (127.10.98.1:35023)
I20260926 12:35:09.546485 22162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85fbc31ba61d41a7a889e25a895cc11e" candidate_uuid: "ec8c24d5e388486eacb880ecce669007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "08438c15d8e6447d8522d3dfe57b2fa5"
I20260926 12:35:09.546626 22162 raft_consensus.cc:3037] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:35:09.547967 22162 raft_consensus.cc:2454] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8c24d5e388486eacb880ecce669007 in term 1.
I20260926 12:35:09.548426 22248 leader_election.cc:302] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 08438c15d8e6447d8522d3dfe57b2fa5, ec8c24d5e388486eacb880ecce669007; no voters:
I20260926 12:35:09.548679 22387 raft_consensus.cc:2781] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:35:09.548904 22387 raft_consensus.cc:686] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 [term 1 LEADER]: Becoming Leader. State: Replica: ec8c24d5e388486eacb880ecce669007, State: Running, Role: LEADER
I20260926 12:35:09.549329 22387 consensus_queue.cc:237] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08438c15d8e6447d8522d3dfe57b2fa5" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37777 } } peers { permanent_uuid: "ea3ab5c1b9d9462391bbe5c9c17bf893" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35023 } } peers { permanent_uuid: "ec8c24d5e388486eacb880ecce669007" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 35827 } }
I20260926 12:35:09.557570 22362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:09.536698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:35023 (local address 127.10.98.3:40099)
0926 12:35:09.542652 (+ 5954us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:09.542667 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:35:09.542760 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:35:09.550027 (+ 7267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:35:09.550031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:35:09.550057 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:35:09.550157 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:09.550164 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.551762 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.551766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:09.552814 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:09.552827 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.557077 (+ 4250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.557081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:09.557201 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:09.557226 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 12:35:09.557290 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 12:35:09.557389 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4381,"thread_start_us":143,"threads_started":1}
I20260926 12:35:09.558089 22376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:09.536089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:40099 (local address 127.10.98.1:35023)
0926 12:35:09.536612 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:09.536616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:09.542912 (+ 6296us) server_negotiation.cc:407] Connection header received
0926 12:35:09.542970 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:09.542972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:09.543027 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:09.543093 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:35:09.550323 (+ 7230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.551105 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.556077 (+ 4972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.556732 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.556827 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:09.556902 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:09.557876 (+ 974us) server_negotiation.cc:299] Negotiation successful
0926 12:35:09.557971 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436}
I20260926 12:35:09.562103 21875 catalog_manager.cc:5495] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 reported cstate change: term changed from 0 to 1, leader changed from <none> to ec8c24d5e388486eacb880ecce669007 (127.10.98.3). New cstate: current_term: 1 leader_uuid: "ec8c24d5e388486eacb880ecce669007" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "08438c15d8e6447d8522d3dfe57b2fa5" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea3ab5c1b9d9462391bbe5c9c17bf893" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ec8c24d5e388486eacb880ecce669007" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 35827 } health_report { overall_health: HEALTHY } } }
I20260926 12:35:09.566437 22029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85fbc31ba61d41a7a889e25a895cc11e" candidate_uuid: "ec8c24d5e388486eacb880ecce669007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3ab5c1b9d9462391bbe5c9c17bf893" is_pre_election: true
I20260926 12:35:09.566776 22029 raft_consensus.cc:2454] T 85fbc31ba61d41a7a889e25a895cc11e P ea3ab5c1b9d9462391bbe5c9c17bf893 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ec8c24d5e388486eacb880ecce669007 in term 0.
I20260926 12:35:09.567314 22042 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85fbc31ba61d41a7a889e25a895cc11e" candidate_uuid: "ec8c24d5e388486eacb880ecce669007" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea3ab5c1b9d9462391bbe5c9c17bf893"
I20260926 12:35:09.567441 22042 raft_consensus.cc:3037] T 85fbc31ba61d41a7a889e25a895cc11e P ea3ab5c1b9d9462391bbe5c9c17bf893 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:35:09.568709 22042 raft_consensus.cc:2454] T 85fbc31ba61d41a7a889e25a895cc11e P ea3ab5c1b9d9462391bbe5c9c17bf893 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ec8c24d5e388486eacb880ecce669007 in term 1.
W20260926 12:35:09.625823 22091 tablet.cc:2369] T 85fbc31ba61d41a7a889e25a895cc11e P ea3ab5c1b9d9462391bbe5c9c17bf893: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:35:09.628747 22378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:09.622782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50736 (local address 127.10.98.3:35827)
0926 12:35:09.623218 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:09.623224 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:09.623238 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:35:09.623330 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:09.623333 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:09.623401 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:09.623524 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:35:09.624024 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.624758 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.626519 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:09.626865 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:09.626963 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:09.627381 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:35:09.627940 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:35:09.628049 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:09.628490 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 12:35:09.628557 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346}
I20260926 12:35:09.640166 22042 raft_consensus.cc:1261] T 85fbc31ba61d41a7a889e25a895cc11e P ea3ab5c1b9d9462391bbe5c9c17bf893 [term 1 FOLLOWER]: Refusing update from remote peer ec8c24d5e388486eacb880ecce669007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:35:09.642396 22387 consensus_queue.cc:1035] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea3ab5c1b9d9462391bbe5c9c17bf893" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 35023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:35:09.650398 22162 raft_consensus.cc:1261] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5 [term 1 FOLLOWER]: Refusing update from remote peer ec8c24d5e388486eacb880ecce669007: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:35:09.651137 22387 consensus_queue.cc:1035] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 [LEADER]: Connected to new peer: Peer: permanent_uuid: "08438c15d8e6447d8522d3dfe57b2fa5" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:35:09.664629 22397 mvcc.cc:205] Tried to move back new op lower bound from 7333585345074630656 to 7333585344730173440. Current Snapshot: MvccSnapshot[applied={T|T < 7333585345074630656}]
I20260926 12:35:09.675885 22394 mvcc.cc:205] Tried to move back new op lower bound from 7333585345074630656 to 7333585344730173440. Current Snapshot: MvccSnapshot[applied={T|T < 7333585345074630656 or (T in {7333585345074630656})}]
I20260926 12:35:09.676450 22395 mvcc.cc:205] Tried to move back new op lower bound from 7333585345074630656 to 7333585344730173440. Current Snapshot: MvccSnapshot[applied={T|T < 7333585345074630656 or (T in {7333585345074630656})}]
I20260926 12:35:09.676668 21876 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:35:09.676853 21876 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45512:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:35:09.679589 21876 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3e77ac6652b94caba18b1ae2afdd28ef: Sending DeleteTablet for 3 replicas of tablet 85fbc31ba61d41a7a889e25a895cc11e
I20260926 12:35:09.680405 22142 tablet_service.cc:1514] Processing DeleteTablet for tablet 85fbc31ba61d41a7a889e25a895cc11e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:35:09 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:48066
I20260926 12:35:09.681478 22024 tablet_service.cc:1514] Processing DeleteTablet for tablet 85fbc31ba61d41a7a889e25a895cc11e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:35:09 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:36678
I20260926 12:35:09.681872 22294 tablet_service.cc:1514] Processing DeleteTablet for tablet 85fbc31ba61d41a7a889e25a895cc11e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:35:09 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:50720
I20260926 12:35:09.682039 22404 tablet_replica.cc:330] T 85fbc31ba61d41a7a889e25a895cc11e P ea3ab5c1b9d9462391bbe5c9c17bf893: stopping tablet replica
I20260926 12:35:09.683210 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 21957
W20260926 12:35:09.691664 21860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.98.1:35023 (error 108)
I20260926 12:35:09.693341 22403 tablet_replica.cc:330] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5: stopping tablet replica
I20260926 12:35:09.693732 22403 raft_consensus.cc:2229] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:35:09.694144 22403 raft_consensus.cc:2258] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:35:09.695958 22403 ts_tablet_manager.cc:1905] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:35:09.696296 22405 tablet_replica.cc:330] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007: stopping tablet replica
I20260926 12:35:09.696771 22405 raft_consensus.cc:2229] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:35:09.697253 22405 raft_consensus.cc:2258] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:35:09.698428 22373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:09.698061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:35023 (local address 127.0.0.1:36684)
0926 12:35:09.698254 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:09.698326 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.1:35023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20260926 12:35:09.698741 21860 catalog_manager.cc:4584] TS ea3ab5c1b9d9462391bbe5c9c17bf893 (127.10.98.1:35023): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 85fbc31ba61d41a7a889e25a895cc11e: Network error: Client connection negotiation failed: client connection to 127.10.98.1:35023: connect: Connection refused (error 111)
I20260926 12:35:09.698861 22403 ts_tablet_manager.cc:1918] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:35:09.698957 22403 log.cc:1196] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/85fbc31ba61d41a7a889e25a895cc11e
I20260926 12:35:09.699251 22403 ts_tablet_manager.cc:1939] T 85fbc31ba61d41a7a889e25a895cc11e P 08438c15d8e6447d8522d3dfe57b2fa5: Deleting consensus metadata
I20260926 12:35:09.700742 22405 ts_tablet_manager.cc:1905] T 85fbc31ba61d41a7a889e25a895cc11e P ec8c24d5e388486eacb880ecce669007: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:35:09.701289 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 22096
I20260926 12:35:09.708535 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 22231
I20260926 12:35:09.709211 22373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:09.708873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:37777 (local address 127.0.0.1:48070)
0926 12:35:09.709043 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:09.709115 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.2:37777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260926 12:35:09.718879 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 21844
I20260926 12:35:09.726392 10632 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 12:35:10.061376 10632 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 12:35:10.061506 10632 mini_ranger.cc:66] Stopping Ranger...
I20260926 12:35:10.396796 10632 mini_ranger.cc:68] Stopped Ranger
2026-09-26 12:35:10.397 UTC [21430] LOG: received smart shutdown request
2026-09-26 12:35:10.400 UTC [21430] LOG: background worker "logical replication launcher" (PID 21439) exited with exit code 1
2026-09-26 12:35:10.401 UTC [21434] LOG: shutting down
2026-09-26 12:35:10.615 UTC [21430] LOG: database system is shut down
2026-09-26T12:35:10Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47723 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:35:10 dist-test-slave-t7tk krb5kdc[22415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:35:10 dist-test-slave-t7tk krb5kdc[22415](info): set up 2 sockets
Sep 26 12:35:10 dist-test-slave-t7tk krb5kdc[22415](info): commencing operation
krb5kdc: starting...
W20260926 12:35:11.687099 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:35:11 dist-test-slave-t7tk krb5kdc[22415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:35:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:35:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.10.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.10.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.10.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/rangeradmin_127.10.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.10.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/rangeradmin_127.10.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.10.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.10.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.10.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/rangerlookup_127.10.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.10.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/rangerlookup_127.10.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.10.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/HTTP_127.10.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/HTTP_127.10.98.20@KRBTEST.COM.keytab.
I20260926 12:35:11.862196 10632 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskFMA2CE/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/postgres -l logfile start
2026-09-26 12:35:14.622 UTC [22445] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 12:35:14.622 UTC [22445] LOG: listening on IPv4 address "127.10.98.20", port 60919
2026-09-26 12:35:14.625 UTC [22445] LOG: listening on Unix socket "/tmp/.s.PGSQL.60919"
2026-09-26 12:35:14.655 UTC [22448] LOG: database system was shut down at 2026-09-26 12:35:12 UTC
2026-09-26 12:35:14.659 UTC [22445] LOG: database system is ready to accept connections
I20260926 12:35:15.526727 10632 mini_postgres.cc:96] Postgres bound to 60919
2026-09-26 12:35:15.532 UTC [22456] FATAL: database "slave" does not exist
127.10.98.20:60919 - accepting connections
I20260926 12:35:15.533286 10632 mini_ranger.cc:161] Starting Ranger...
I20260926 12:35:15.547902 10632 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 12:35:16.062017 10632 mini_ranger.cc:87] Created ranger Postgres database
I20260926 12:35:16.062117 10632 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ranger-admin
2026-09-26 12:35:16,331 [I] DB FLAVOR :POSTGRES
2026-09-26 12:35:16,331 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 12:35:16,331 [I] Checking connection..
2026-09-26 12:35:16,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 12:35:16,562 [I] Checking connection passed.
2026-09-26 12:35:16,563 [I] --------- Verifying version history table ---------
2026-09-26 12:35:16,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 12:35:16,807 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 12:35:16,808 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 12:35:17,060 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 12:35:17,061 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 12:35:17,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 12:35:17.286 UTC [22520] WARNING: there is no transaction in progress
2026-09-26 12:35:17,292 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 12:35:17,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 12:35:17,527 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 12:35:17,527 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 12:35:17,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 12:35:17,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 12:35:18,016 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-t7tk.c.gcp-upstream.internal','N') ;"
2026-09-26 12:35:18,241 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 12:35:18,485 [I] Table x_portal_user does not exist in database ranger
2026-09-26 12:35:18,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 12:35:18,747 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 12:35:18,748 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 12:35:19,010 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 12:35:19,274 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 12:35:19,274 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 12:35:20.332 UTC [22628] WARNING: there is no transaction in progress
2026-09-26 12:35:20.351 UTC [22628] WARNING: there is no transaction in progress
2026-09-26 12:35:20.369 UTC [22628] WARNING: there is no transaction in progress
2026-09-26 12:35:20.386 UTC [22628] WARNING: there is no transaction in progress
2026-09-26 12:35:20.406 UTC [22628] WARNING: there is no transaction in progress
2026-09-26 12:35:20.426 UTC [22628] WARNING: there is no transaction in progress
2026-09-26 12:35:20.490 UTC [22628] WARNING: there is no transaction in progress
2026-09-26 12:35:20.499 UTC [22628] WARNING: there is no transaction in progress
2026-09-26 12:35:20.517 UTC [22628] WARNING: there is no transaction in progress
2026-09-26 12:35:20.530 UTC [22628] WARNING: there is no transaction in progress
2026-09-26 12:35:20.544 UTC [22628] WARNING: there is no transaction in progress
2026-09-26 12:35:20.561 UTC [22628] WARNING: there is no transaction in progress
2026-09-26 12:35:20.573 UTC [22628] WARNING: there is no transaction in progress
2026-09-26 12:35:20.582 UTC [22628] WARNING: there is no transaction in progress
2026-09-26 12:35:20.590 UTC [22628] WARNING: there is no transaction in progress
2026-09-26 12:35:20.598 UTC [22628] WARNING: there is no transaction in progress
2026-09-26 12:35:21,355 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 12:35:21,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 12:35:21,597 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 12:35:21,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 12:35:21,864 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 12:35:21,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 12:35:22,129 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 12:35:22,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 12:35:22,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 12:35:22,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-t7tk.c.gcp-upstream.internal';"
2026-09-26 12:35:22,870 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 12:35:22,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 12:35:23,106 [I] DB_PATCHES have already been applied
I20260926 12:35:23.111634 10632 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ranger-admin:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskFMA2CE/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 12:35:23.111720 10632 mini_ranger.cc:193] Using host: 127.10.98.20
I20260926 12:35:23.115057 10632 mini_ranger.cc:227] Ranger admin URL: http://127.10.98.20:60311
Sep 26, 2026 12:35:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 12:35:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 12:35:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 12:35:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 12:35:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 12:35:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 12:35:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 12:35:24.133471 10632 mini_ranger.cc:160] Time spent starting Ranger: real 8.600s user 0.009s sys 0.001s
Sep 26, 2026 12:35:44 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 12:35:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 26, 2026 12:35:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 12:35:44 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 12:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 12:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 12:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 12:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 12:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 12:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 12:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 12:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 12:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 12:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 12:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 12:35:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 12:35:44 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 12:35:45 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 12:35:45.642699 10632 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.10.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.10.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.10.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/rangerkms_127.10.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.10.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/rangerkms_127.10.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.10.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.10.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.10.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/HTTP_127.10.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.10.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/HTTP_127.10.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 12:35:45 dist-test-slave-t7tk krb5kdc[22415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426145, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 12:35:45.736163 10632 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 12:35:45.746136 10632 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 12:35:46.766000 10632 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 12:35:46.766072 10632 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ranger-kms
I20260926 12:35:46.766089 10632 mini_ranger_kms.cc:227] Using postgres at 127.10.98.20:60919
2026-09-26 12:35:46,920 [I] DB FLAVOR :POSTGRES
2026-09-26 12:35:46,921 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 12:35:46,921 [I] Checking connection
2026-09-26 12:35:46,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 12:35:47,157 [I] connection success
2026-09-26 12:35:47,158 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 12:35:47,158 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 12:35:47,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 12:35:47,397 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 12:35:47,397 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 12:35:47,397 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 12:35:47,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.10.98.20:60919/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 12:35:47,675 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 12:35:48.158622 10632 mini_ranger_kms.cc:310] Created kms service
I20260926 12:35:48.370926 10632 mini_ranger_kms.cc:326] Created kudu user
I20260926 12:35:48.419627 10632 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 12:35:48 dist-test-slave-t7tk krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426145, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
I20260926 12:35:48.805275 10632 mini_ranger_kms.cc:384] Added ranger policy
I20260926 12:35:48.805433 10632 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ranger-kms:/tmp/dist-test-taskFMA2CE/build/debug/bin/postgresql.jar:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskFMA2CE/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 12:35:48.805457 10632 mini_ranger_kms.cc:240] Using host: 127.10.98.20
I20260926 12:35:48.809643 10632 mini_ranger_kms.cc:276] Ranger KMS PID: 22808
I20260926 12:35:48.809732 10632 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.10.98.20:40233
Sep 26, 2026 12:35:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 12:35:49 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 12:35:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 12:35:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 12:35:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 12:35:50 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 12:35:50.926898 10632 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.191s user 0.013s sys 0.127s
I20260926 12:35:50.927034 10632 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 12:35:50.927103 10632 mini_ranger_kms.cc:401] 127.10.98.20:40233/kms/v1/keys
Sep 26 12:35:51 dist-test-slave-t7tk krb5kdc[22415](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790426151, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 12:35:51 dist-test-slave-t7tk krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426151, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.98.20@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
Sep 26 12:35:51 dist-test-slave-t7tk krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426151, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.10.98.20@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
Sep 26 12:35:54 dist-test-slave-t7tk krb5kdc[22415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:35:54.120615 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:35941
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:35941
--ranger_config_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.98.20:40233/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:35:54.232616 22859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:35:54.232908 22859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:35:54.232949 22859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:35:54.236996 22859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:35:54.237058 22859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:35:54.237077 22859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:35:54.237092 22859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:35:54.237123 22859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:35:54.242684 22859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/wal
--ranger_kms_url=127.10.98.20:40233/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:35941
--ranger_config_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:35941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:35:54.243700 22859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:35:54.244700 22859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:35:54.253273 22867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:35:54.253283 22865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:35:54.253338 22859 server_base.cc:1034] running on GCE node
W20260926 12:35:54.253285 22864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:35:54.254127 22859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:35:54.255285 22859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:35:54.256445 22859 hybrid_clock.cc:648] HybridClock initialized: now 1790426154256432 us; error 26 us; skew 500 ppm
Sep 26 12:35:54 dist-test-slave-t7tk krb5kdc[22415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426154, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:35:54.265911 22859 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:35:54.267011 22859 webserver.cc:460] Webserver started at http://127.10.98.62:37869/ using document root <none> and password file <none>
I20260926 12:35:54.267565 22859 fs_manager.cc:359] Metadata directory not provided
I20260926 12:35:54.267627 22859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:35:54.267807 22859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:35:54 dist-test-slave-t7tk krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426154, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
I20260926 12:35:54.324594 22859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "19d10d878a5a456daadcd2f3ec365746"
format_stamp: "Formatted at 2026-09-26 12:35:54 on dist-test-slave-t7tk"
server_key: "e74f53aaf06dd0947d9401289958b7b9"
server_key_iv: "6e2511e230e8d2dccd9fce366ebfaaf4"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:54.325366 22859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "19d10d878a5a456daadcd2f3ec365746"
format_stamp: "Formatted at 2026-09-26 12:35:54 on dist-test-slave-t7tk"
server_key: "e74f53aaf06dd0947d9401289958b7b9"
server_key_iv: "6e2511e230e8d2dccd9fce366ebfaaf4"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:54.330617 22859 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260926 12:35:54.349749 22874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:54.351301 22859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:35:54.351459 22859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "19d10d878a5a456daadcd2f3ec365746"
format_stamp: "Formatted at 2026-09-26 12:35:54 on dist-test-slave-t7tk"
server_key: "e74f53aaf06dd0947d9401289958b7b9"
server_key_iv: "6e2511e230e8d2dccd9fce366ebfaaf4"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:54.351586 22859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:35:54.367925 22859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:35:54.371300 22859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:35:54.371507 22859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:35:54.378885 22859 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:35941
I20260926 12:35:54.378930 22926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:35941 every 8 connection(s)
I20260926 12:35:54.379894 22859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:35:54.383596 22927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:35:54.388478 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 22859
I20260926 12:35:54.388609 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:35:54.388891 10632 external_mini_cluster.cc:1428] Setting key cd657980da47fabe57be2b02b3729d93
I20260926 12:35:54.391322 22927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746: Bootstrap starting.
I20260926 12:35:54.394367 22927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746: Neither blocks nor log segments found. Creating new log.
I20260926 12:35:54.395315 22927 log.cc:824] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746: Log is configured to *not* fsync() on all Append() calls
I20260926 12:35:54.397447 22927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746: No bootstrap required, opened a new log
Sep 26 12:35:54 dist-test-slave-t7tk krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:35:54.401400 22927 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d10d878a5a456daadcd2f3ec365746" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35941 } }
I20260926 12:35:54.401712 22927 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:35:54.401791 22927 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d10d878a5a456daadcd2f3ec365746, State: Initialized, Role: FOLLOWER
I20260926 12:35:54.402269 22927 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d10d878a5a456daadcd2f3ec365746" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35941 } }
I20260926 12:35:54.402473 22927 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:35:54.402565 22927 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:35:54.402693 22927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:35:54.403919 22927 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d10d878a5a456daadcd2f3ec365746" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35941 } }
I20260926 12:35:54.404285 22927 leader_election.cc:302] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19d10d878a5a456daadcd2f3ec365746; no voters:
I20260926 12:35:54.404420 22930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:54.390491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38226 (local address 127.10.98.62:35941)
0926 12:35:54.390908 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:54.390924 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:54.390963 (+ 39us) server_negotiation.cc:407] Connection header received
0926 12:35:54.391647 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:54.391667 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:54.392018 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:54.392330 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:35:54.393166 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:54.394141 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:54.395049 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:54.395574 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:54.395718 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:54.398671 (+ 2953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:35:54.398678 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:35:54.398698 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:35:54.398730 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:35:54.400607 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:54.401175 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:54.401182 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:54.401188 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:54.401324 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:54.401626 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:54.401629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:54.401631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:54.402181 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:35:54.402398 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:54.403131 (+ 733us) server_negotiation.cc:299] Negotiation successful
0926 12:35:54.403426 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":115,"threads_started":1}
I20260926 12:35:54.404614 22927 leader_election.cc:288] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:35:54.404901 22932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:35:54.405164 22932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [term 1 LEADER]: Becoming Leader. State: Replica: 19d10d878a5a456daadcd2f3ec365746, State: Running, Role: LEADER
I20260926 12:35:54.405905 22927 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:35:54.405529 22932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d10d878a5a456daadcd2f3ec365746" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35941 } }
I20260926 12:35:54.407842 22934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19d10d878a5a456daadcd2f3ec365746. Latest consensus state: current_term: 1 leader_uuid: "19d10d878a5a456daadcd2f3ec365746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d10d878a5a456daadcd2f3ec365746" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35941 } } }
I20260926 12:35:54.407976 22934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [sys.catalog]: This master's current role is: LEADER
I20260926 12:35:54.408386 22933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19d10d878a5a456daadcd2f3ec365746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d10d878a5a456daadcd2f3ec365746" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35941 } } }
I20260926 12:35:54.408529 22933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [sys.catalog]: This master's current role is: LEADER
I20260926 12:35:54.408514 22927 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 12:35:54.409313 22946 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:35:54.415652 22946 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:35:54.423789 22946 catalog_manager.cc:1347] Generated new cluster ID: f97af0bc90914e51832fc7eea7652565
I20260926 12:35:54.423864 22946 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:35:54.445662 22946 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:35:54.446995 22946 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:35:54.463449 22946 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746: Generated new TSK 0
I20260926 12:35:54.464406 22946 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
I20260926 12:35:55.115694 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.98.20:40233/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:35941
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:35:55.232407 22972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:35:55.232679 22972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:35:55.232738 22972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:35:55.237246 22972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:35:55.237322 22972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:35:55.237468 22972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:35:55.243052 22972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.10.98.20:40233/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:35941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22972
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:35:55.244053 22972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:35:55.245016 22972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:35:55.256052 22977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:35:55.256304 22972 server_base.cc:1034] running on GCE node
W20260926 12:35:55.256353 22978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:35:55.256474 22980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:35:55.256949 22972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:35:55.257858 22972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:35:55.259459 22972 hybrid_clock.cc:648] HybridClock initialized: now 1790426155259424 us; error 49 us; skew 500 ppm
Sep 26 12:35:55 dist-test-slave-t7tk krb5kdc[22415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426155, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:35:55.263759 22972 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:35:55.264755 22972 webserver.cc:460] Webserver started at http://127.10.98.1:35569/ using document root <none> and password file <none>
I20260926 12:35:55.265533 22972 fs_manager.cc:359] Metadata directory not provided
I20260926 12:35:55.265627 22972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:35:55.265854 22972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:35:55 dist-test-slave-t7tk krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426155, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
I20260926 12:35:55.282317 22972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "881dfe725ce84948ba5a1109c1b5ce8d"
format_stamp: "Formatted at 2026-09-26 12:35:55 on dist-test-slave-t7tk"
server_key: "43abf5fdf9c3410ca5a5a33a2792e3db"
server_key_iv: "020aae27ab0db47d4762b8502e04e0d0"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:55.283125 22972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "881dfe725ce84948ba5a1109c1b5ce8d"
format_stamp: "Formatted at 2026-09-26 12:35:55 on dist-test-slave-t7tk"
server_key: "43abf5fdf9c3410ca5a5a33a2792e3db"
server_key_iv: "020aae27ab0db47d4762b8502e04e0d0"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:55.290328 22972 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260926 12:35:55.309454 22989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:55.311172 22972 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260926 12:35:55.311307 22972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "881dfe725ce84948ba5a1109c1b5ce8d"
format_stamp: "Formatted at 2026-09-26 12:35:55 on dist-test-slave-t7tk"
server_key: "43abf5fdf9c3410ca5a5a33a2792e3db"
server_key_iv: "020aae27ab0db47d4762b8502e04e0d0"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:55.311456 22972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:35:55.325846 22972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:35:55.329444 22972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:35:55.329679 22972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:35:55.330471 22972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:35:55.331610 22972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:35:55.331707 22972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:55.331791 22972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:35:55.331841 22972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:55.345536 22972 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:39707
I20260926 12:35:55.346834 22972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:35:55.348620 23102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:39707 every 8 connection(s)
I20260926 12:35:55.353752 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 22972
I20260926 12:35:55.353874 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:35:55.354188 10632 external_mini_cluster.cc:1428] Setting key 6981dfd7d3e96b268f8f89100db8c9f1
WARNING: no policy specified for kudu/127.10.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.2@KRBTEST.COM" created.
Sep 26 12:35:55 dist-test-slave-t7tk krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426155, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:35:55.375442 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:55.352341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:43471 (local address 127.10.98.62:35941)
0926 12:35:55.361355 (+ 9014us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:55.361364 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:55.361384 (+ 20us) server_negotiation.cc:407] Connection header received
0926 12:35:55.361438 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:55.361442 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:55.361496 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:55.361606 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:35:55.363491 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:55.364394 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:55.366260 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:55.366639 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:55.366748 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:55.369899 (+ 3151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:35:55.369903 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:35:55.369906 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:35:55.369937 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:35:55.371794 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:55.372475 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:55.372478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:55.372480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:55.372578 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:55.373020 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:55.373023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:55.373025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:55.373304 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:35:55.373406 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:55.375104 (+ 1698us) server_negotiation.cc:299] Negotiation successful
0926 12:35:55.375246 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8905,"thread_start_us":112,"threads_started":1}
I20260926 12:35:55.376791 23105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:55.351154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:35941 (local address 127.10.98.1:43471)
0926 12:35:55.351767 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:55.351831 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0926 12:35:55.352915 (+ 1084us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:35:55.362035 (+ 9120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:35:55.362048 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:35:55.362461 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:35:55.363284 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:55.363309 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:55.364565 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:55.364569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:55.366104 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:55.366118 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:55.366960 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:55.366964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:55.367178 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:55.367844 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:35:55.367866 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:35:55.369662 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:35:55.371962 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:35:55.371969 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:35:55.371988 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:35:55.372345 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:35:55.372679 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:35:55.372682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:35:55.372684 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:35:55.372899 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:35:55.373464 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:35:55.373471 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:35:55.373842 (+ 371us) client_negotiation.cc:769] Sending connection context
0926 12:35:55.374091 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 12:35:55.374462 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":135,"threads_started":1}
I20260926 12:35:55.378539 23103 heartbeater.cc:347] Connected to a master server at 127.10.98.62:35941
I20260926 12:35:55.378800 23103 heartbeater.cc:464] Registering TS with master...
I20260926 12:35:55.379396 23103 heartbeater.cc:511] Master 127.10.98.62:35941 requested a full tablet report, sending...
I20260926 12:35:55.381313 22891 ts_manager.cc:194] Registered new tserver with Master: 881dfe725ce84948ba5a1109c1b5ce8d (127.10.98.1:39707)
I20260926 12:35:55.382920 22891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:43471
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
I20260926 12:35:55.398242 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.98.20:40233/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:35941
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:35:55 dist-test-slave-t7tk krb5kdc[22415](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790426155, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 12:35:55.530583 23111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:35:55.530848 23111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:35:55.530908 23111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:35:55.536504 23111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:35:55.536583 23111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:35:55.536665 23111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:35:55.542498 23111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.10.98.20:40233/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:35941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:35:55.543622 23111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:35:55.544647 23111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:35:55.559201 23120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:35:55.559219 23117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:35:55.559751 23111 server_base.cc:1034] running on GCE node
W20260926 12:35:55.559906 23118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:35:55.560339 23111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:35:55.561028 23111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:35:55.562202 23111 hybrid_clock.cc:648] HybridClock initialized: now 1790426155562170 us; error 43 us; skew 500 ppm
Sep 26 12:35:55 dist-test-slave-t7tk krb5kdc[22415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426155, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:35:55.565017 23111 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:35:55.565984 23111 webserver.cc:460] Webserver started at http://127.10.98.2:42011/ using document root <none> and password file <none>
I20260926 12:35:55.566533 23111 fs_manager.cc:359] Metadata directory not provided
I20260926 12:35:55.566601 23111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:35:55.566843 23111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:35:55 dist-test-slave-t7tk krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426155, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
I20260926 12:35:55.582751 23111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "d6d44f49d58c4771bc117698b63f033e"
format_stamp: "Formatted at 2026-09-26 12:35:55 on dist-test-slave-t7tk"
server_key: "1058fa4f4837495acafc30c8ee5a1f84"
server_key_iv: "d33c868600bf0781d29b1e8bd3e20536"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:55.583487 23111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "d6d44f49d58c4771bc117698b63f033e"
format_stamp: "Formatted at 2026-09-26 12:35:55 on dist-test-slave-t7tk"
server_key: "1058fa4f4837495acafc30c8ee5a1f84"
server_key_iv: "d33c868600bf0781d29b1e8bd3e20536"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:55.590250 23111 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260926 12:35:55.608881 23127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:55.610411 23111 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.001s
I20260926 12:35:55.610541 23111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "d6d44f49d58c4771bc117698b63f033e"
format_stamp: "Formatted at 2026-09-26 12:35:55 on dist-test-slave-t7tk"
server_key: "1058fa4f4837495acafc30c8ee5a1f84"
server_key_iv: "d33c868600bf0781d29b1e8bd3e20536"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:55.610673 23111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:35:55.628759 23111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:35:55.632210 23111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:35:55.632468 23111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:35:55.633234 23111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:35:55.634385 23111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:35:55.634496 23111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:55.634596 23111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:35:55.634665 23111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:55.648444 23111 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:37901
I20260926 12:35:55.649587 23111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:35:55.649566 23240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:37901 every 8 connection(s)
I20260926 12:35:55.658754 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 23111
Sep 26 12:35:55 dist-test-slave-t7tk krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426155, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:35:55.658888 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:35:55.659198 10632 external_mini_cluster.cc:1428] Setting key 3a72d065621d6370e0d61ae2c47035ae
I20260926 12:35:55.663395 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:55.652937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:44285 (local address 127.10.98.62:35941)
0926 12:35:55.653053 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:55.653058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:55.653074 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:35:55.653625 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:55.653628 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:55.653682 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:55.653755 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:35:55.654837 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:55.655431 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:55.656603 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:55.656826 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:55.656916 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:55.659444 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:35:55.659448 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:35:55.659450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:35:55.659482 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:35:55.661020 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:55.661639 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:55.661642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:55.661644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:55.661725 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:55.662091 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:55.662094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:55.662095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:55.662291 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:35:55.662372 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:55.663089 (+ 717us) server_negotiation.cc:299] Negotiation successful
0926 12:35:55.663222 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:35:55.665000 23243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:55.651885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:35941 (local address 127.10.98.2:44285)
0926 12:35:55.652362 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:55.652415 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 12:35:55.653412 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:35:55.653944 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:35:55.653951 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:35:55.654217 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:35:55.654697 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:55.654710 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:55.655548 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:55.655552 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:55.656496 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:55.656505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:55.656963 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:55.656967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:55.657197 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:55.657739 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:35:55.657755 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:35:55.659236 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:35:55.661174 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:35:55.661180 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:35:55.661195 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:35:55.661531 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:35:55.661824 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:35:55.661827 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:35:55.661829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:35:55.661994 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:35:55.662434 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:35:55.662441 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:35:55.662805 (+ 364us) client_negotiation.cc:769] Sending connection context
0926 12:35:55.663606 (+ 801us) client_negotiation.cc:240] Negotiation successful
0926 12:35:55.663957 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":117,"threads_started":1}
I20260926 12:35:55.666357 23241 heartbeater.cc:347] Connected to a master server at 127.10.98.62:35941
I20260926 12:35:55.666607 23241 heartbeater.cc:464] Registering TS with master...
I20260926 12:35:55.667192 23241 heartbeater.cc:511] Master 127.10.98.62:35941 requested a full tablet report, sending...
I20260926 12:35:55.668315 22891 ts_manager.cc:194] Registered new tserver with Master: d6d44f49d58c4771bc117698b63f033e (127.10.98.2:37901)
I20260926 12:35:55.668890 22891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:44285
WARNING: no policy specified for kudu/127.10.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
I20260926 12:35:55.692080 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.98.20:40233/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:35941
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:35:55.842413 23246 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:35:55.842864 23246 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:35:55.843014 23246 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:35:55.849093 23246 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:35:55.849198 23246 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:35:55.849282 23246 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:35:55.857164 23246 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.10.98.20:40233/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:35941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23246
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:35:55.858565 23246 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:35:55.859855 23246 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:35:55.872784 23252 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:35:55.872871 23251 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:35:55.873116 23254 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:35:55.873523 23246 server_base.cc:1034] running on GCE node
I20260926 12:35:55.874217 23246 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:35:55.874930 23246 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:35:55.876137 23246 hybrid_clock.cc:648] HybridClock initialized: now 1790426155876104 us; error 40 us; skew 500 ppm
Sep 26 12:35:55 dist-test-slave-t7tk krb5kdc[22415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426155, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:35:55.879027 23246 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:35:55.880067 23246 webserver.cc:460] Webserver started at http://127.10.98.3:44969/ using document root <none> and password file <none>
I20260926 12:35:55.880739 23246 fs_manager.cc:359] Metadata directory not provided
I20260926 12:35:55.880807 23246 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:35:55.881024 23246 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 12:35:55 dist-test-slave-t7tk krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426155, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
Sep 26 12:35:55 dist-test-slave-t7tk krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426155, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
I20260926 12:35:55.899113 23246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "9c82a4810ff8419c94aa237b7123e7c8"
format_stamp: "Formatted at 2026-09-26 12:35:55 on dist-test-slave-t7tk"
server_key: "49d708d2db4318437a5242db5f3a0648"
server_key_iv: "e4c64f1162089d9775cfed071309d37c"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:55.900271 23246 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "9c82a4810ff8419c94aa237b7123e7c8"
format_stamp: "Formatted at 2026-09-26 12:35:55 on dist-test-slave-t7tk"
server_key: "49d708d2db4318437a5242db5f3a0648"
server_key_iv: "e4c64f1162089d9775cfed071309d37c"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:55.907718 23246 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260926 12:35:55.933585 23261 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:55.938076 23246 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.001s
I20260926 12:35:55.938256 23246 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "9c82a4810ff8419c94aa237b7123e7c8"
format_stamp: "Formatted at 2026-09-26 12:35:55 on dist-test-slave-t7tk"
server_key: "49d708d2db4318437a5242db5f3a0648"
server_key_iv: "e4c64f1162089d9775cfed071309d37c"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:55.938429 23246 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:35:55.968537 23246 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:35:55.971503 23246 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:35:55.971668 23246 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:35:55.972290 23246 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:35:55.973176 23246 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:35:55.973224 23246 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:55.973269 23246 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:35:55.973299 23246 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:55.983211 23246 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:37949
I20260926 12:35:55.983244 23374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:37949 every 8 connection(s)
I20260926 12:35:55.984112 23246 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:35:55.991536 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 23246
I20260926 12:35:55.991655 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:35:55.992275 10632 external_mini_cluster.cc:1428] Setting key 63fd22f8f1693269507868f175102c62
Sep 26 12:35:55 dist-test-slave-t7tk krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426155, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:35:55.997778 23106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:55.986092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:57725 (local address 127.10.98.62:35941)
0926 12:35:55.986250 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:55.986255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:55.987121 (+ 866us) server_negotiation.cc:407] Connection header received
0926 12:35:55.988095 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:55.988098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:55.988155 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:55.988228 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:35:55.989325 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:55.989845 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:55.991016 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:55.991233 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:55.991321 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:55.994190 (+ 2869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:35:55.994194 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:35:55.994214 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:35:55.994245 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:35:55.995724 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:55.996215 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:55.996219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:55.996221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:55.996299 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:55.996623 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:55.996627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:55.996629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:55.996812 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:35:55.996902 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:55.997496 (+ 594us) server_negotiation.cc:299] Negotiation successful
0926 12:35:55.997618 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:35:55.998778 23377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:55.986499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:35941 (local address 127.10.98.3:57725)
0926 12:35:55.986975 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:55.987027 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 12:35:55.987892 (+ 865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:35:55.988411 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:35:55.988417 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:35:55.988675 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:35:55.989195 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:55.989210 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:55.989961 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:55.989964 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:55.990900 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:55.990910 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:55.991507 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:55.991511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:55.991703 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:55.992305 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:35:55.992325 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:35:55.993990 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:35:55.995835 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:35:55.995841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:35:55.995867 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:35:55.996121 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:35:55.996384 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:35:55.996387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:35:55.996389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:35:55.996539 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:35:55.996916 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:35:55.996924 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:35:55.997253 (+ 329us) client_negotiation.cc:769] Sending connection context
0926 12:35:55.997440 (+ 187us) client_negotiation.cc:240] Negotiation successful
0926 12:35:55.997771 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":118,"threads_started":1}
I20260926 12:35:55.999977 23375 heartbeater.cc:347] Connected to a master server at 127.10.98.62:35941
I20260926 12:35:56.000233 23375 heartbeater.cc:464] Registering TS with master...
I20260926 12:35:56.000821 23375 heartbeater.cc:511] Master 127.10.98.62:35941 requested a full tablet report, sending...
I20260926 12:35:56.002063 22891 ts_manager.cc:194] Registered new tserver with Master: 9c82a4810ff8419c94aa237b7123e7c8 (127.10.98.3:37949)
I20260926 12:35:56.002835 22891 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:57725
I20260926 12:35:56.006873 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:35:56.007663 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 22972
I20260926 12:35:56.014521 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 23111
I20260926 12:35:56.021019 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 23246
I20260926 12:35:56.026297 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 22859
I20260926 12:35:56.032024 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:35941
--webserver_interface=127.10.98.62
--webserver_port=37869
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:35941
--ranger_config_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.98.20:40233/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.98.20:40233/kms with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:35:56.142273 23378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:35:56.142582 23378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:35:56.142623 23378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:35:56.146570 23378 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:35:56.146629 23378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:35:56.146646 23378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:35:56.146661 23378 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:35:56.146674 23378 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:35:56.152129 23378 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.98.20:40233/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:35941
--ranger_config_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:35941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=37869
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:35:56.153193 23378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:35:56.154225 23378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:35:56.163050 23386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:35:56.163064 23383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:35:56.163077 23378 server_base.cc:1034] running on GCE node
W20260926 12:35:56.163076 23384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:35:56.163856 23378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:35:56.165032 23378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:35:56.166191 23378 hybrid_clock.cc:648] HybridClock initialized: now 1790426156166166 us; error 41 us; skew 500 ppm
Sep 26 12:35:56 dist-test-slave-t7tk krb5kdc[22415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426156, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:35:56.168988 23378 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:35:56.169957 23378 webserver.cc:460] Webserver started at http://127.10.98.62:37869/ using document root <none> and password file <none>
I20260926 12:35:56.170460 23378 fs_manager.cc:359] Metadata directory not provided
I20260926 12:35:56.170522 23378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 12:35:56 dist-test-slave-t7tk krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426156, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
I20260926 12:35:56.237305 23378 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:35:56.239755 23394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:56.241252 23378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:35:56.241382 23378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "19d10d878a5a456daadcd2f3ec365746"
format_stamp: "Formatted at 2026-09-26 12:35:54 on dist-test-slave-t7tk"
server_key: "e74f53aaf06dd0947d9401289958b7b9"
server_key_iv: "6e2511e230e8d2dccd9fce366ebfaaf4"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:56.241954 23378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:35:56.260123 23378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:35:56.263082 23378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:35:56.263262 23378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:35:56.270443 23378 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:35941
I20260926 12:35:56.270464 23446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:35941 every 8 connection(s)
I20260926 12:35:56.271502 23378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:35:56.275775 23447 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:35:56.277472 23447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746: Bootstrap starting.
I20260926 12:35:56.279552 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 23378
I20260926 12:35:56.279963 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.98.20:40233/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:39707
--local_ip_for_outbound_sockets=127.10.98.1
--tserver_master_addrs=127.10.98.62:35941
--webserver_port=35569
--webserver_interface=127.10.98.1
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.98.20:40233/kms with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:35:56.293706 23447 log.cc:824] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746: Log is configured to *not* fsync() on all Append() calls
I20260926 12:35:56.300411 23447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:35:56.300797 23447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746: Bootstrap complete.
I20260926 12:35:56.304802 23447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d10d878a5a456daadcd2f3ec365746" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35941 } }
I20260926 12:35:56.305230 23447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19d10d878a5a456daadcd2f3ec365746, State: Initialized, Role: FOLLOWER
I20260926 12:35:56.305704 23447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d10d878a5a456daadcd2f3ec365746" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35941 } }
I20260926 12:35:56.305816 23447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:35:56.305879 23447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:35:56.305965 23447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [term 1 FOLLOWER]: Advancing to term 2
I20260926 12:35:56.307792 23447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d10d878a5a456daadcd2f3ec365746" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35941 } }
I20260926 12:35:56.308097 23447 leader_election.cc:302] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19d10d878a5a456daadcd2f3ec365746; no voters:
I20260926 12:35:56.308382 23447 leader_election.cc:288] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 12:35:56.308527 23451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 12:35:56.308887 23451 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [term 2 LEADER]: Becoming Leader. State: Replica: 19d10d878a5a456daadcd2f3ec365746, State: Running, Role: LEADER
I20260926 12:35:56.309304 23451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d10d878a5a456daadcd2f3ec365746" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35941 } }
I20260926 12:35:56.309583 23447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:35:56.310832 23452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "19d10d878a5a456daadcd2f3ec365746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d10d878a5a456daadcd2f3ec365746" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35941 } } }
I20260926 12:35:56.310858 23453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19d10d878a5a456daadcd2f3ec365746. Latest consensus state: current_term: 2 leader_uuid: "19d10d878a5a456daadcd2f3ec365746" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19d10d878a5a456daadcd2f3ec365746" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35941 } } }
I20260926 12:35:56.310950 23452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [sys.catalog]: This master's current role is: LEADER
I20260926 12:35:56.310954 23453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746 [sys.catalog]: This master's current role is: LEADER
I20260926 12:35:56.311733 23447 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 12:35:56.312595 23462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:35:56.318897 23462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:35:56.319777 23462 catalog_manager.cc:1259] Loaded cluster ID: f97af0bc90914e51832fc7eea7652565
I20260926 12:35:56.319839 23462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:35:56.322504 23462 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:35:56.323254 23462 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746: Loaded TSK: 0
I20260926 12:35:56.324764 23462 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 12:35:56.397256 23449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:35:56.397528 23449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:35:56.397593 23449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:35:56.401793 23449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:35:56.401852 23449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:35:56.401938 23449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:35:56.408255 23449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.98.20:40233/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:39707
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=35569
--tserver_master_addrs=127.10.98.62:35941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:35:56.409480 23449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:35:56.410534 23449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:35:56.423151 23484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:35:56.423169 23487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:35:56.423722 23485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:35:56.424388 23449 server_base.cc:1034] running on GCE node
I20260926 12:35:56.424896 23449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:35:56.425686 23449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:35:56.426867 23449 hybrid_clock.cc:648] HybridClock initialized: now 1790426156426837 us; error 40 us; skew 500 ppm
Sep 26 12:35:56 dist-test-slave-t7tk krb5kdc[22415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426156, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:35:56.429889 23449 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:35:56.431130 23449 webserver.cc:460] Webserver started at http://127.10.98.1:35569/ using document root <none> and password file <none>
I20260926 12:35:56.431866 23449 fs_manager.cc:359] Metadata directory not provided
I20260926 12:35:56.431972 23449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 12:35:56 dist-test-slave-t7tk krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426156, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
I20260926 12:35:56.464331 23449 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:35:56.476606 23495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:56.481040 23449 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.002s sys 0.000s
I20260926 12:35:56.481245 23449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "881dfe725ce84948ba5a1109c1b5ce8d"
format_stamp: "Formatted at 2026-09-26 12:35:55 on dist-test-slave-t7tk"
server_key: "43abf5fdf9c3410ca5a5a33a2792e3db"
server_key_iv: "020aae27ab0db47d4762b8502e04e0d0"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:56.482071 23449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:35:56.502261 23449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:35:56.506960 23449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:35:56.507157 23449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:35:56.507789 23449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:35:56.508749 23449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:35:56.508792 23449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:56.508841 23449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:35:56.508860 23449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:56.518965 23449 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:39707
I20260926 12:35:56.520292 23449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:35:56.521137 23608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:39707 every 8 connection(s)
I20260926 12:35:56.523671 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 23449
I20260926 12:35:56.524200 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.98.20:40233/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:37901
--local_ip_for_outbound_sockets=127.10.98.2
--tserver_master_addrs=127.10.98.62:35941
--webserver_port=42011
--webserver_interface=127.10.98.2
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.98.20:40233/kms with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:35:56 dist-test-slave-t7tk krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426156, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:35:56.547740 23611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:56.522994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:35941 (local address 127.10.98.1:58967)
0926 12:35:56.524478 (+ 1484us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:56.524541 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0926 12:35:56.525638 (+ 1097us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:35:56.529734 (+ 4096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:35:56.529744 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:35:56.530096 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:35:56.530807 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:56.530830 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:56.532771 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:56.532774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:56.534186 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:56.534198 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:56.535502 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:56.535507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:56.535705 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:56.536372 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:35:56.536393 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:35:56.538220 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:35:56.543427 (+ 5207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:35:56.543434 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:35:56.543452 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:35:56.543843 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:35:56.544294 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:35:56.544298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:35:56.544299 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:35:56.544504 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:35:56.545419 (+ 915us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:35:56.545426 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:35:56.545882 (+ 456us) client_negotiation.cc:769] Sending connection context
0926 12:35:56.546073 (+ 191us) client_negotiation.cc:240] Negotiation successful
0926 12:35:56.546514 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1045,"thread_start_us":134,"threads_started":1}
I20260926 12:35:56.550537 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:56.526135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:58967 (local address 127.10.98.62:35941)
0926 12:35:56.528093 (+ 1958us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:56.528114 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:56.528159 (+ 45us) server_negotiation.cc:407] Connection header received
0926 12:35:56.528458 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:56.528473 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:56.528880 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:56.529652 (+ 772us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:35:56.531017 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:56.532608 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:56.534333 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:56.535091 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:56.535255 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:56.538498 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:35:56.538507 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:35:56.538526 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:35:56.538563 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:35:56.543265 (+ 4702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:56.543997 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:56.544004 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:56.544012 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:56.544176 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:56.544670 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:56.544674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:56.544676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:56.545248 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:35:56.548061 (+ 2813us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:56.549209 (+ 1148us) server_negotiation.cc:299] Negotiation successful
0926 12:35:56.549536 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1798,"thread_start_us":123,"threads_started":1}
I20260926 12:35:56.553369 23609 heartbeater.cc:347] Connected to a master server at 127.10.98.62:35941
I20260926 12:35:56.553637 23609 heartbeater.cc:464] Registering TS with master...
I20260926 12:35:56.554266 23609 heartbeater.cc:511] Master 127.10.98.62:35941 requested a full tablet report, sending...
W20260926 12:35:56.556648 23609 heartbeater.cc:644] Failed to heartbeat to 127.10.98.62:35941 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 12:35:56.556782 23609 heartbeater.cc:464] Registering TS with master...
I20260926 12:35:56.556982 23609 heartbeater.cc:511] Master 127.10.98.62:35941 requested a full tablet report, sending...
I20260926 12:35:56.557849 23609 heartbeater.cc:464] Registering TS with master...
I20260926 12:35:56.558045 23609 heartbeater.cc:511] Master 127.10.98.62:35941 requested a full tablet report, sending...
W20260926 12:35:56.558801 23609 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:35:56.674615 23612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:35:56.675030 23612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:35:56.675112 23612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:35:56.681571 23612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:35:56.681658 23612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:35:56.681746 23612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:35:56.689958 23612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.98.20:40233/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:37901
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=42011
--tserver_master_addrs=127.10.98.62:35941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:35:56.691531 23612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:35:56.693037 23612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:35:56.705561 23621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:35:56.705561 23619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:35:56.706228 23618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:35:56.707110 23612 server_base.cc:1034] running on GCE node
I20260926 12:35:56.707584 23612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:35:56.708371 23612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:35:56.709563 23612 hybrid_clock.cc:648] HybridClock initialized: now 1790426156709532 us; error 49 us; skew 500 ppm
Sep 26 12:35:56 dist-test-slave-t7tk krb5kdc[22415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426156, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:35:56.713758 23612 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:35:56.714916 23612 webserver.cc:460] Webserver started at http://127.10.98.2:42011/ using document root <none> and password file <none>
I20260926 12:35:56.715651 23612 fs_manager.cc:359] Metadata directory not provided
I20260926 12:35:56.715754 23612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 12:35:56 dist-test-slave-t7tk krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426156, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
I20260926 12:35:56.741394 23612 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:35:56.744398 23629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:56.746050 23612 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:35:56.746176 23612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "d6d44f49d58c4771bc117698b63f033e"
format_stamp: "Formatted at 2026-09-26 12:35:55 on dist-test-slave-t7tk"
server_key: "1058fa4f4837495acafc30c8ee5a1f84"
server_key_iv: "d33c868600bf0781d29b1e8bd3e20536"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:56.746908 23612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:35:56.765874 23612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:35:56.770148 23612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:35:56.770561 23612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:35:56.771647 23612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:35:56.772773 23612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:35:56.772835 23612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:56.772891 23612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:35:56.772915 23612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:56.783695 23612 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:37901
I20260926 12:35:56.783946 23742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:37901 every 8 connection(s)
I20260926 12:35:56.785061 23612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:35:56.786130 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 23612
I20260926 12:35:56.786665 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.10.98.20:40233/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:37949
--local_ip_for_outbound_sockets=127.10.98.3
--tserver_master_addrs=127.10.98.62:35941
--webserver_port=44969
--webserver_interface=127.10.98.3
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:38933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.10.98.20:40233/kms with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:35:56 dist-test-slave-t7tk krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426156, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:35:56.819015 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:56.791518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:50371 (local address 127.10.98.62:35941)
0926 12:35:56.791679 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:56.791685 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:56.805812 (+ 14127us) server_negotiation.cc:407] Connection header received
0926 12:35:56.806826 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:56.806830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:56.806902 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:56.807021 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:35:56.808607 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:56.809468 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:56.811238 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:56.811600 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:56.811681 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:56.814843 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:35:56.814847 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:35:56.814850 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:35:56.814881 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:35:56.816511 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:56.817170 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:56.817174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:56.817176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:56.817267 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:56.817682 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:56.817686 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:56.817688 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:56.817917 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:35:56.818001 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:56.818700 (+ 699us) server_negotiation.cc:299] Negotiation successful
0926 12:35:56.818843 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 12:35:56.820830 23746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:56.790286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:35941 (local address 127.10.98.2:50371)
0926 12:35:56.805547 (+ 15261us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:56.805640 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0926 12:35:56.806589 (+ 949us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:35:56.807269 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:35:56.807281 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:35:56.807659 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:35:56.808420 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:56.808441 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:56.809636 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:56.809640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:56.811095 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:56.811107 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:56.811904 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:56.811908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:56.812113 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:56.812898 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:35:56.812920 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:35:56.814648 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:35:56.816646 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:35:56.816653 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:35:56.816670 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:35:56.817020 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:35:56.817389 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:35:56.817392 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:35:56.817393 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:35:56.817569 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:35:56.818072 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:35:56.818079 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:35:56.818418 (+ 339us) client_negotiation.cc:769] Sending connection context
0926 12:35:56.819508 (+ 1090us) client_negotiation.cc:240] Negotiation successful
0926 12:35:56.819847 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14920,"thread_start_us":149,"threads_started":1}
I20260926 12:35:56.822697 23743 heartbeater.cc:347] Connected to a master server at 127.10.98.62:35941
I20260926 12:35:56.822939 23743 heartbeater.cc:464] Registering TS with master...
I20260926 12:35:56.823524 23743 heartbeater.cc:511] Master 127.10.98.62:35941 requested a full tablet report, sending...
W20260926 12:35:56.825238 23743 heartbeater.cc:644] Failed to heartbeat to 127.10.98.62:35941 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 12:35:56.825366 23743 heartbeater.cc:464] Registering TS with master...
I20260926 12:35:56.825554 23743 heartbeater.cc:511] Master 127.10.98.62:35941 requested a full tablet report, sending...
I20260926 12:35:56.826596 23743 heartbeater.cc:464] Registering TS with master...
I20260926 12:35:56.826761 23743 heartbeater.cc:511] Master 127.10.98.62:35941 requested a full tablet report, sending...
W20260926 12:35:56.827553 23743 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:35:56.932711 23745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:35:56.933023 23745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:35:56.933066 23745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:35:56.937386 23745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:35:56.937458 23745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:35:56.937541 23745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:35:56.944358 23745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:38933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.10.98.20:40233/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:37949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=44969
--tserver_master_addrs=127.10.98.62:35941
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:35:56.945817 23745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:35:56.947283 23745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:35:56.959743 23753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:35:56.959719 23752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:35:56.959725 23755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:35:56.961077 23745 server_base.cc:1034] running on GCE node
I20260926 12:35:56.961578 23745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:35:56.962369 23745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:35:56.963574 23745 hybrid_clock.cc:648] HybridClock initialized: now 1790426156963541 us; error 49 us; skew 500 ppm
Sep 26 12:35:56 dist-test-slave-t7tk krb5kdc[22415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426156, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:35:56.967793 23745 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:35:56.968950 23745 webserver.cc:460] Webserver started at http://127.10.98.3:44969/ using document root <none> and password file <none>
I20260926 12:35:56.969719 23745 fs_manager.cc:359] Metadata directory not provided
I20260926 12:35:56.969810 23745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 12:35:56 dist-test-slave-t7tk krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426156, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
I20260926 12:35:56.994303 23745 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 12:35:56.997789 23763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:56.999631 23745 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260926 12:35:56.999850 23745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "9c82a4810ff8419c94aa237b7123e7c8"
format_stamp: "Formatted at 2026-09-26 12:35:55 on dist-test-slave-t7tk"
server_key: "49d708d2db4318437a5242db5f3a0648"
server_key_iv: "e4c64f1162089d9775cfed071309d37c"
server_key_version: "kuduclusterkey@0"
I20260926 12:35:57.000641 23745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:35:57.026018 23745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:35:57.030076 23745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:35:57.030299 23745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:35:57.031106 23745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:35:57.032198 23745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:35:57.032265 23745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:57.032326 23745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:35:57.032358 23745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:35:57.043723 23745 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:37949
I20260926 12:35:57.044171 23876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:37949 every 8 connection(s)
I20260926 12:35:57.045305 23745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:35:57.046339 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 23745
I20260926 12:35:57.058904 23613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:57.047419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.10.98.62:35941)
0926 12:35:57.047553 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:57.047557 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:57.049763 (+ 2206us) server_negotiation.cc:407] Connection header received
0926 12:35:57.049941 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:57.049945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:57.049997 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:57.050066 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:35:57.051013 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:57.051799 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:57.053051 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:57.053409 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:57.053545 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:57.055415 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:35:57.055419 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:35:57.055422 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:35:57.055449 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:35:57.057023 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:57.057462 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:57.057465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:57.057468 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:57.057561 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:57.058059 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:57.058063 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:57.058064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:57.058242 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:35:57.058441 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:57.058612 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 12:35:57.058729 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":56}
Sep 26 12:35:57 dist-test-slave-t7tk krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426156, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:35:57.065596 23880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:57.049240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:55769 (local address 127.10.98.62:35941)
0926 12:35:57.053300 (+ 4060us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:57.053304 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:57.053318 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:35:57.053354 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:57.053357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:57.053407 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:57.053510 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:35:57.055539 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:57.056355 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:57.058111 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:57.058489 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:57.058588 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:57.061521 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:35:57.061525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:35:57.061528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:35:57.061555 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:35:57.063126 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:57.063797 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:57.063800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:57.063801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:57.063879 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:57.064328 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:57.064331 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:57.064332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:57.064524 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:35:57.064613 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:57.065248 (+ 635us) server_negotiation.cc:299] Negotiation successful
0926 12:35:57.065383 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":133,"server-negotiator.queue_time_us":3977,"thread_start_us":91,"threads_started":1}
I20260926 12:35:57.066473 23881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:57.049418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:35941 (local address 127.10.98.3:55769)
0926 12:35:57.050586 (+ 1168us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:57.050627 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:35:57.051520 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:35:57.054225 (+ 2705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:35:57.054236 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:35:57.054624 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:35:57.055346 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:57.055371 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:57.056524 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:57.056528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:57.057974 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:57.057985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:57.058759 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:57.058764 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:57.059179 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:57.059777 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:35:57.059797 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:35:57.061334 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:35:57.063259 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:35:57.063266 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:35:57.063285 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:35:57.063679 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:35:57.063969 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:35:57.063972 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:35:57.063974 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:35:57.064213 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:35:57.064661 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:35:57.064667 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:35:57.064951 (+ 284us) client_negotiation.cc:769] Sending connection context
0926 12:35:57.065191 (+ 240us) client_negotiation.cc:240] Negotiation successful
0926 12:35:57.065530 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"thread_start_us":132,"threads_started":1}
I20260926 12:35:57.067593 23877 heartbeater.cc:347] Connected to a master server at 127.10.98.62:35941
I20260926 12:35:57.067847 23877 heartbeater.cc:464] Registering TS with master...
I20260926 12:35:57.068394 23877 heartbeater.cc:511] Master 127.10.98.62:35941 requested a full tablet report, sending...
W20260926 12:35:57.069864 23877 heartbeater.cc:644] Failed to heartbeat to 127.10.98.62:35941 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 12:35:57.069980 23877 heartbeater.cc:464] Registering TS with master...
I20260926 12:35:57.070155 23877 heartbeater.cc:511] Master 127.10.98.62:35941 requested a full tablet report, sending...
I20260926 12:35:57.071059 23877 heartbeater.cc:464] Registering TS with master...
I20260926 12:35:57.071240 23877 heartbeater.cc:511] Master 127.10.98.62:35941 requested a full tablet report, sending...
W20260926 12:35:57.072227 23877 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 12:35:57 dist-test-slave-t7tk krb5kdc[22415](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790426157, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:35:57.559019 23609 heartbeater.cc:464] Registering TS with master...
I20260926 12:35:57.559541 23609 heartbeater.cc:511] Master 127.10.98.62:35941 requested a full tablet report, sending...
I20260926 12:35:57.561388 23410 ts_manager.cc:194] Registered new tserver with Master: 881dfe725ce84948ba5a1109c1b5ce8d (127.10.98.1:39707)
I20260926 12:35:57.563211 23410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:58967
Sep 26 12:35:57 dist-test-slave-t7tk krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426157, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
I20260926 12:35:57.827786 23743 heartbeater.cc:464] Registering TS with master...
I20260926 12:35:57.828076 23743 heartbeater.cc:511] Master 127.10.98.62:35941 requested a full tablet report, sending...
I20260926 12:35:57.828884 23410 ts_manager.cc:194] Registered new tserver with Master: d6d44f49d58c4771bc117698b63f033e (127.10.98.2:37901)
I20260926 12:35:57.829656 23410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:50371
I20260926 12:35:58.072460 23877 heartbeater.cc:464] Registering TS with master...
I20260926 12:35:58.072796 23877 heartbeater.cc:511] Master 127.10.98.62:35941 requested a full tablet report, sending...
I20260926 12:35:58.073763 23410 ts_manager.cc:194] Registered new tserver with Master: 9c82a4810ff8419c94aa237b7123e7c8 (127.10.98.3:37949)
I20260926 12:35:58.074558 23410 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:55769
I20260926 12:35:58.077960 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:35:58.096342 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:58.079926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38234 (local address 127.10.98.62:35941)
0926 12:35:58.080533 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:58.080538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:58.080793 (+ 255us) server_negotiation.cc:407] Connection header received
0926 12:35:58.080941 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:58.080944 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:58.080995 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:58.081085 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:35:58.082065 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.082903 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.084108 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.084436 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.084535 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:58.086363 (+ 1828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:35:58.086367 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:35:58.086370 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:35:58.086410 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:35:58.093858 (+ 7448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:58.094480 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:58.094484 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:58.094486 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:58.094589 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:35:58.094913 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:35:58.094916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:35:58.094918 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:35:58.095160 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:35:58.095285 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:58.096025 (+ 740us) server_negotiation.cc:299] Negotiation successful
0926 12:35:58.096160 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":76,"threads_started":1}
I20260926 12:35:58.100723 23410 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38234:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:35:58.104984 23410 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Sep 26 12:35:58 dist-test-slave-t7tk krb5kdc[22415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426157, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for HTTP/127.10.98.20@KRBTEST.COM
I20260926 12:35:58.128309 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:58.121495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:37901 (local address 127.0.0.1:53618)
0926 12:35:58.122650 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:58.122663 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:35:58.122764 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:35:58.123494 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:35:58.123499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:35:58.123521 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:35:58.123650 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:58.123657 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.125083 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.125087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:58.126659 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:58.126671 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.127761 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.127765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:58.127869 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:58.127910 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 12:35:58.128013 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 12:35:58.128143 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1095,"thread_start_us":80,"threads_started":1}
I20260926 12:35:58.128962 23909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:58.121745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53618 (local address 127.10.98.2:37901)
0926 12:35:58.122497 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:58.122504 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:58.122869 (+ 365us) server_negotiation.cc:407] Connection header received
0926 12:35:58.122956 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:58.122962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:58.123177 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:58.123317 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:35:58.123771 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.124950 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.126764 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.127452 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.127544 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:58.127704 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:58.128705 (+ 1001us) server_negotiation.cc:299] Negotiation successful
0926 12:35:58.128822 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":94,"threads_started":1}
I20260926 12:35:58.132293 23664 tablet_service.cc:1467] Processing CreateTablet for tablet 94d0b496fb494165ad89af60f16fb938 (DEFAULT_TABLE table=test-table [id=3940553880b24271bb1c54ca9bcdac2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:35:58.133610 23664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d0b496fb494165ad89af60f16fb938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:35:58.135085 23908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:58.120802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:37949 (local address 127.0.0.1:46034)
0926 12:35:58.122017 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:58.122072 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 12:35:58.122208 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:35:58.127599 (+ 5391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:35:58.127602 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:35:58.127621 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:35:58.127715 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:58.127720 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.131119 (+ 3399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.131123 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:58.132274 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:58.132282 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.134676 (+ 2394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.134680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:58.134774 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:58.134801 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 12:35:58.134847 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 12:35:58.134940 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1020,"thread_start_us":109,"threads_started":1}
I20260926 12:35:58.135762 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:58.120927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46034 (local address 127.10.98.3:37949)
0926 12:35:58.127089 (+ 6162us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:58.127097 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:58.127125 (+ 28us) server_negotiation.cc:407] Connection header received
0926 12:35:58.127206 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:58.127212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:58.127439 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:58.127601 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:35:58.128336 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.131000 (+ 2664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.132435 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.134302 (+ 1867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.135347 (+ 1045us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:58.135468 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:58.135520 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 12:35:58.135618 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6041,"thread_start_us":108,"threads_started":1}
I20260926 12:35:58.135994 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:58.121335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47810 (local address 127.10.98.1:39707)
0926 12:35:58.121736 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:58.121743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:58.128628 (+ 6885us) server_negotiation.cc:407] Connection header received
0926 12:35:58.128727 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:58.128732 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:58.128957 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:58.129128 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:35:58.130869 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.132040 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.134426 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.135193 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.135276 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:58.135439 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:58.135764 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 12:35:58.135866 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":93,"threads_started":1}
I20260926 12:35:58.136320 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:58.120810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:39707 (local address 127.0.0.1:47810)
0926 12:35:58.128494 (+ 7684us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:58.128506 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:35:58.128579 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:35:58.130661 (+ 2082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:35:58.130664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:35:58.130680 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:35:58.130752 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:58.130756 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.132165 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.132168 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:58.134266 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:58.134278 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.135531 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.135534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:58.135613 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:58.135634 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:35:58.136119 (+ 485us) client_negotiation.cc:240] Negotiation successful
0926 12:35:58.136194 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7631,"mutex_wait_us":1190,"thread_start_us":90,"threads_started":1}
I20260926 12:35:58.138833 23811 tablet_service.cc:1467] Processing CreateTablet for tablet 94d0b496fb494165ad89af60f16fb938 (DEFAULT_TABLE table=test-table [id=3940553880b24271bb1c54ca9bcdac2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:35:58.139366 23543 tablet_service.cc:1467] Processing CreateTablet for tablet 94d0b496fb494165ad89af60f16fb938 (DEFAULT_TABLE table=test-table [id=3940553880b24271bb1c54ca9bcdac2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:35:58.139987 23811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d0b496fb494165ad89af60f16fb938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:35:58.140336 23543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d0b496fb494165ad89af60f16fb938. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:35:58.150754 23913 tablet_bootstrap.cc:492] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d: Bootstrap starting.
I20260926 12:35:58.151620 23912 tablet_bootstrap.cc:492] T 94d0b496fb494165ad89af60f16fb938 P 9c82a4810ff8419c94aa237b7123e7c8: Bootstrap starting.
I20260926 12:35:58.153223 23913 tablet_bootstrap.cc:654] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d: Neither blocks nor log segments found. Creating new log.
I20260926 12:35:58.154130 23913 log.cc:824] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d: Log is configured to *not* fsync() on all Append() calls
I20260926 12:35:58.154865 23912 tablet_bootstrap.cc:654] T 94d0b496fb494165ad89af60f16fb938 P 9c82a4810ff8419c94aa237b7123e7c8: Neither blocks nor log segments found. Creating new log.
I20260926 12:35:58.155608 23912 log.cc:824] T 94d0b496fb494165ad89af60f16fb938 P 9c82a4810ff8419c94aa237b7123e7c8: Log is configured to *not* fsync() on all Append() calls
I20260926 12:35:58.157979 23913 tablet_bootstrap.cc:492] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d: No bootstrap required, opened a new log
I20260926 12:35:58.158216 23913 ts_tablet_manager.cc:1397] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260926 12:35:58.158424 23912 tablet_bootstrap.cc:492] T 94d0b496fb494165ad89af60f16fb938 P 9c82a4810ff8419c94aa237b7123e7c8: No bootstrap required, opened a new log
I20260926 12:35:58.158617 23912 ts_tablet_manager.cc:1397] T 94d0b496fb494165ad89af60f16fb938 P 9c82a4810ff8419c94aa237b7123e7c8: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 12:35:58.162775 23912 raft_consensus.cc:348] T 94d0b496fb494165ad89af60f16fb938 P 9c82a4810ff8419c94aa237b7123e7c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c82a4810ff8419c94aa237b7123e7c8" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 37949 } } peers { permanent_uuid: "881dfe725ce84948ba5a1109c1b5ce8d" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 39707 } } peers { permanent_uuid: "d6d44f49d58c4771bc117698b63f033e" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37901 } }
I20260926 12:35:58.162776 23913 raft_consensus.cc:348] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c82a4810ff8419c94aa237b7123e7c8" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 37949 } } peers { permanent_uuid: "881dfe725ce84948ba5a1109c1b5ce8d" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 39707 } } peers { permanent_uuid: "d6d44f49d58c4771bc117698b63f033e" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37901 } }
I20260926 12:35:58.163087 23912 raft_consensus.cc:374] T 94d0b496fb494165ad89af60f16fb938 P 9c82a4810ff8419c94aa237b7123e7c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:35:58.163087 23913 raft_consensus.cc:374] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:35:58.163146 23912 raft_consensus.cc:729] T 94d0b496fb494165ad89af60f16fb938 P 9c82a4810ff8419c94aa237b7123e7c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c82a4810ff8419c94aa237b7123e7c8, State: Initialized, Role: FOLLOWER
I20260926 12:35:58.163146 23913 raft_consensus.cc:729] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 881dfe725ce84948ba5a1109c1b5ce8d, State: Initialized, Role: FOLLOWER
I20260926 12:35:58.163687 23913 consensus_queue.cc:260] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c82a4810ff8419c94aa237b7123e7c8" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 37949 } } peers { permanent_uuid: "881dfe725ce84948ba5a1109c1b5ce8d" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 39707 } } peers { permanent_uuid: "d6d44f49d58c4771bc117698b63f033e" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37901 } }
I20260926 12:35:58.163687 23912 consensus_queue.cc:260] T 94d0b496fb494165ad89af60f16fb938 P 9c82a4810ff8419c94aa237b7123e7c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c82a4810ff8419c94aa237b7123e7c8" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 37949 } } peers { permanent_uuid: "881dfe725ce84948ba5a1109c1b5ce8d" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 39707 } } peers { permanent_uuid: "d6d44f49d58c4771bc117698b63f033e" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37901 } }
I20260926 12:35:58.164840 23912 ts_tablet_manager.cc:1428] T 94d0b496fb494165ad89af60f16fb938 P 9c82a4810ff8419c94aa237b7123e7c8: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
I20260926 12:35:58.164840 23913 ts_tablet_manager.cc:1428] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260926 12:35:58.165043 23911 tablet_bootstrap.cc:492] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e: Bootstrap starting.
I20260926 12:35:58.165246 23877 heartbeater.cc:503] Master 127.10.98.62:35941 was elected leader, sending a full tablet report...
I20260926 12:35:58.166530 23609 heartbeater.cc:503] Master 127.10.98.62:35941 was elected leader, sending a full tablet report...
I20260926 12:35:58.178124 23911 tablet_bootstrap.cc:654] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e: Neither blocks nor log segments found. Creating new log.
I20260926 12:35:58.179232 23911 log.cc:824] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e: Log is configured to *not* fsync() on all Append() calls
I20260926 12:35:58.182694 23911 tablet_bootstrap.cc:492] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e: No bootstrap required, opened a new log
I20260926 12:35:58.182909 23911 ts_tablet_manager.cc:1397] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260926 12:35:58.186471 23911 raft_consensus.cc:348] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c82a4810ff8419c94aa237b7123e7c8" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 37949 } } peers { permanent_uuid: "881dfe725ce84948ba5a1109c1b5ce8d" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 39707 } } peers { permanent_uuid: "d6d44f49d58c4771bc117698b63f033e" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37901 } }
I20260926 12:35:58.186820 23911 raft_consensus.cc:374] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:35:58.186878 23911 raft_consensus.cc:729] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6d44f49d58c4771bc117698b63f033e, State: Initialized, Role: FOLLOWER
I20260926 12:35:58.187408 23911 consensus_queue.cc:260] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c82a4810ff8419c94aa237b7123e7c8" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 37949 } } peers { permanent_uuid: "881dfe725ce84948ba5a1109c1b5ce8d" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 39707 } } peers { permanent_uuid: "d6d44f49d58c4771bc117698b63f033e" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37901 } }
I20260926 12:35:58.188691 23911 ts_tablet_manager.cc:1428] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:35:58.189090 23743 heartbeater.cc:503] Master 127.10.98.62:35941 was elected leader, sending a full tablet report...
I20260926 12:35:58.219126 23916 raft_consensus.cc:482] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:35:58.219350 23916 raft_consensus.cc:504] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c82a4810ff8419c94aa237b7123e7c8" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 37949 } } peers { permanent_uuid: "881dfe725ce84948ba5a1109c1b5ce8d" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 39707 } } peers { permanent_uuid: "d6d44f49d58c4771bc117698b63f033e" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37901 } }
I20260926 12:35:58.220547 23916 leader_election.cc:288] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c82a4810ff8419c94aa237b7123e7c8 (127.10.98.3:37949), d6d44f49d58c4771bc117698b63f033e (127.10.98.2:37901)
I20260926 12:35:58.227851 23909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:58.221163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:54369 (local address 127.10.98.2:37901)
0926 12:35:58.221282 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:58.221287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:58.223082 (+ 1795us) server_negotiation.cc:407] Connection header received
0926 12:35:58.223264 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:58.223267 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:58.223320 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:58.223366 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:35:58.223793 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.224475 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.226551 (+ 2076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.227096 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.227179 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:58.227251 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:58.227611 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 12:35:58.227710 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:35:58.228307 23921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:58.220800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:37901 (local address 127.10.98.1:54369)
0926 12:35:58.222998 (+ 2198us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:58.223011 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:35:58.223147 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:35:58.223511 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:35:58.223514 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:35:58.223530 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:35:58.223663 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:58.223671 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.224619 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.225282 (+ 663us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:58.226405 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:58.226416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.227352 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.227355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:58.227463 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:58.227487 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:35:58.228073 (+ 586us) client_negotiation.cc:240] Negotiation successful
0926 12:35:58.228152 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2131,"thread_start_us":71,"threads_started":1}
I20260926 12:35:58.229022 23696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0b496fb494165ad89af60f16fb938" candidate_uuid: "881dfe725ce84948ba5a1109c1b5ce8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6d44f49d58c4771bc117698b63f033e" is_pre_election: true
I20260926 12:35:58.229388 23696 raft_consensus.cc:2454] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 881dfe725ce84948ba5a1109c1b5ce8d in term 0.
I20260926 12:35:58.229933 23499 leader_election.cc:302] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 881dfe725ce84948ba5a1109c1b5ce8d, d6d44f49d58c4771bc117698b63f033e; no voters:
I20260926 12:35:58.230270 23916 raft_consensus.cc:2781] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:35:58.230357 23916 raft_consensus.cc:482] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:35:58.230398 23916 raft_consensus.cc:3037] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:35:58.231752 23916 raft_consensus.cc:504] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c82a4810ff8419c94aa237b7123e7c8" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 37949 } } peers { permanent_uuid: "881dfe725ce84948ba5a1109c1b5ce8d" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 39707 } } peers { permanent_uuid: "d6d44f49d58c4771bc117698b63f033e" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37901 } }
I20260926 12:35:58.232143 23916 leader_election.cc:288] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d [CANDIDATE]: Term 1 election: Requested vote from peers 9c82a4810ff8419c94aa237b7123e7c8 (127.10.98.3:37949), d6d44f49d58c4771bc117698b63f033e (127.10.98.2:37901)
I20260926 12:35:58.232584 23696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0b496fb494165ad89af60f16fb938" candidate_uuid: "881dfe725ce84948ba5a1109c1b5ce8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6d44f49d58c4771bc117698b63f033e"
I20260926 12:35:58.232728 23696 raft_consensus.cc:3037] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:35:58.234010 23696 raft_consensus.cc:2454] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 881dfe725ce84948ba5a1109c1b5ce8d in term 1.
I20260926 12:35:58.234330 23499 leader_election.cc:302] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 881dfe725ce84948ba5a1109c1b5ce8d, d6d44f49d58c4771bc117698b63f033e; no voters:
I20260926 12:35:58.234521 23916 raft_consensus.cc:2781] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:35:58.234771 23916 raft_consensus.cc:686] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d [term 1 LEADER]: Becoming Leader. State: Replica: 881dfe725ce84948ba5a1109c1b5ce8d, State: Running, Role: LEADER
I20260926 12:35:58.235157 23916 consensus_queue.cc:237] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c82a4810ff8419c94aa237b7123e7c8" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 37949 } } peers { permanent_uuid: "881dfe725ce84948ba5a1109c1b5ce8d" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 39707 } } peers { permanent_uuid: "d6d44f49d58c4771bc117698b63f033e" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37901 } }
I20260926 12:35:58.238587 23411 catalog_manager.cc:5495] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d reported cstate change: term changed from 0 to 1, leader changed from <none> to 881dfe725ce84948ba5a1109c1b5ce8d (127.10.98.1). New cstate: current_term: 1 leader_uuid: "881dfe725ce84948ba5a1109c1b5ce8d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c82a4810ff8419c94aa237b7123e7c8" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 37949 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "881dfe725ce84948ba5a1109c1b5ce8d" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 39707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d6d44f49d58c4771bc117698b63f033e" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37901 } health_report { overall_health: UNKNOWN } } }
I20260926 12:35:58.247987 23906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:58.221163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:39327 (local address 127.10.98.3:37949)
0926 12:35:58.221282 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:58.221287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:58.221592 (+ 305us) server_negotiation.cc:407] Connection header received
0926 12:35:58.221808 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:58.221812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:58.221866 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:58.221914 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:35:58.222363 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.223114 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.246536 (+ 23422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.247179 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.247269 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:58.247343 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:58.247757 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 12:35:58.247847 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:35:58.248389 23922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:58.220728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:37949 (local address 127.10.98.1:39327)
0926 12:35:58.221507 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:58.221522 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:35:58.221676 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:35:58.222074 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:35:58.222077 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:35:58.222094 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:35:58.222240 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:58.222246 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.224724 (+ 2478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.224727 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:58.246323 (+ 21596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:35:58.246341 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.247467 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.247471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:35:58.247587 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:58.247614 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 12:35:58.248145 (+ 531us) client_negotiation.cc:240] Negotiation successful
0926 12:35:58.248238 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":117,"threads_started":1}
I20260926 12:35:58.249078 23831 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0b496fb494165ad89af60f16fb938" candidate_uuid: "881dfe725ce84948ba5a1109c1b5ce8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c82a4810ff8419c94aa237b7123e7c8" is_pre_election: true
I20260926 12:35:58.249065 23830 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d0b496fb494165ad89af60f16fb938" candidate_uuid: "881dfe725ce84948ba5a1109c1b5ce8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c82a4810ff8419c94aa237b7123e7c8"
I20260926 12:35:58.249440 23831 raft_consensus.cc:2454] T 94d0b496fb494165ad89af60f16fb938 P 9c82a4810ff8419c94aa237b7123e7c8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 881dfe725ce84948ba5a1109c1b5ce8d in term 0.
I20260926 12:35:58.271629 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:58.266156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47812 (local address 127.10.98.1:39707)
0926 12:35:58.266349 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 12:35:58.266353 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:35:58.266367 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:35:58.266453 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:35:58.266456 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:35:58.266536 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:35:58.266652 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:35:58.266988 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.267774 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.269330 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:35:58.269680 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:35:58.269782 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:35:58.270229 (+ 447us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:35:58.270823 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:35:58.270942 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 12:35:58.271390 (+ 448us) server_negotiation.cc:299] Negotiation successful
0926 12:35:58.271470 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260926 12:35:58.272789 23610 tablet.cc:2369] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:35:58.287292 23744 tablet.cc:2369] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:35:58.297348 23878 tablet.cc:2369] T 94d0b496fb494165ad89af60f16fb938 P 9c82a4810ff8419c94aa237b7123e7c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:35:58.298700 23696 raft_consensus.cc:1261] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e [term 1 FOLLOWER]: Refusing update from remote peer 881dfe725ce84948ba5a1109c1b5ce8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:35:58.299536 23923 consensus_queue.cc:1035] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6d44f49d58c4771bc117698b63f033e" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:35:58.301163 23830 raft_consensus.cc:3037] T 94d0b496fb494165ad89af60f16fb938 P 9c82a4810ff8419c94aa237b7123e7c8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:35:58.302585 23830 raft_consensus.cc:1261] T 94d0b496fb494165ad89af60f16fb938 P 9c82a4810ff8419c94aa237b7123e7c8 [term 1 FOLLOWER]: Refusing update from remote peer 881dfe725ce84948ba5a1109c1b5ce8d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:35:58.303821 23923 consensus_queue.cc:1035] T 94d0b496fb494165ad89af60f16fb938 P 881dfe725ce84948ba5a1109c1b5ce8d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c82a4810ff8419c94aa237b7123e7c8" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 37949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:35:58.330626 23927 mvcc.cc:205] Tried to move back new op lower bound from 7333585544344453120 to 7333585544134590464. Current Snapshot: MvccSnapshot[applied={T|T < 7333585544344453120}]
I20260926 12:35:58.332196 23411 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:35:58.332367 23411 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38234:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:35:58.335319 23411 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 19d10d878a5a456daadcd2f3ec365746: Sending DeleteTablet for 3 replicas of tablet 94d0b496fb494165ad89af60f16fb938
I20260926 12:35:58.336128 23543 tablet_service.cc:1514] Processing DeleteTablet for tablet 94d0b496fb494165ad89af60f16fb938 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:35:58 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:47810
I20260926 12:35:58.337239 23811 tablet_service.cc:1514] Processing DeleteTablet for tablet 94d0b496fb494165ad89af60f16fb938 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:35:58 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:46034
I20260926 12:35:58.339138 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 23449
I20260926 12:35:58.338173 23664 tablet_service.cc:1514] Processing DeleteTablet for tablet 94d0b496fb494165ad89af60f16fb938 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:35:58 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:53618
I20260926 12:35:58.349339 23936 tablet_replica.cc:330] T 94d0b496fb494165ad89af60f16fb938 P 9c82a4810ff8419c94aa237b7123e7c8: stopping tablet replica
I20260926 12:35:58.349745 23936 raft_consensus.cc:2229] T 94d0b496fb494165ad89af60f16fb938 P 9c82a4810ff8419c94aa237b7123e7c8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:35:58.350127 23936 raft_consensus.cc:2258] T 94d0b496fb494165ad89af60f16fb938 P 9c82a4810ff8419c94aa237b7123e7c8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:35:58.350564 23937 tablet_replica.cc:330] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e: stopping tablet replica
I20260926 12:35:58.350863 23937 raft_consensus.cc:2229] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:35:58.351174 23937 raft_consensus.cc:2258] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:35:58.352849 23936 ts_tablet_manager.cc:1905] T 94d0b496fb494165ad89af60f16fb938 P 9c82a4810ff8419c94aa237b7123e7c8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:35:58.355826 23936 ts_tablet_manager.cc:1918] T 94d0b496fb494165ad89af60f16fb938 P 9c82a4810ff8419c94aa237b7123e7c8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:35:58.355944 23936 log.cc:1196] T 94d0b496fb494165ad89af60f16fb938 P 9c82a4810ff8419c94aa237b7123e7c8: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/94d0b496fb494165ad89af60f16fb938
I20260926 12:35:58.356237 23936 ts_tablet_manager.cc:1939] T 94d0b496fb494165ad89af60f16fb938 P 9c82a4810ff8419c94aa237b7123e7c8: Deleting consensus metadata
I20260926 12:35:58.363911 23937 ts_tablet_manager.cc:1905] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:35:58.366935 23937 ts_tablet_manager.cc:1918] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:35:58.367120 23937 log.cc:1196] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/94d0b496fb494165ad89af60f16fb938
I20260926 12:35:58.367452 23937 ts_tablet_manager.cc:1939] T 94d0b496fb494165ad89af60f16fb938 P d6d44f49d58c4771bc117698b63f033e: Deleting consensus metadata
I20260926 12:35:58.368973 23398 catalog_manager.cc:4841] TS d6d44f49d58c4771bc117698b63f033e (127.10.98.2:37901): tablet 94d0b496fb494165ad89af60f16fb938 (table test-table [id=3940553880b24271bb1c54ca9bcdac2d]) successfully deleted
I20260926 12:35:58.369771 23396 catalog_manager.cc:4841] TS 9c82a4810ff8419c94aa237b7123e7c8 (127.10.98.3:37949): tablet 94d0b496fb494165ad89af60f16fb938 (table test-table [id=3940553880b24271bb1c54ca9bcdac2d]) successfully deleted
W20260926 12:35:58.372347 23397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.98.1:39707 (error 108)
I20260926 12:35:58.373504 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 23612
I20260926 12:35:58.382251 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:35:58.381815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:39707 (local address 127.0.0.1:47816)
0926 12:35:58.382040 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 12:35:58.382142 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.1:39707: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
W20260926 12:35:58.382484 23397 catalog_manager.cc:4584] TS 881dfe725ce84948ba5a1109c1b5ce8d (127.10.98.1:39707): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 94d0b496fb494165ad89af60f16fb938: Network error: Client connection negotiation failed: client connection to 127.10.98.1:39707: connect: Connection refused (error 111)
I20260926 12:35:58.384285 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 23745
I20260926 12:35:58.391523 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 23378
I20260926 12:35:58.399475 10632 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 12:35:58.734426 10632 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 12:35:58.734551 10632 mini_ranger.cc:66] Stopping Ranger...
I20260926 12:35:59.070031 10632 mini_ranger.cc:68] Stopped Ranger
2026-09-26 12:35:59.070 UTC [22445] LOG: received smart shutdown request
2026-09-26 12:35:59.074 UTC [22445] LOG: background worker "logical replication launcher" (PID 22454) exited with exit code 1
2026-09-26 12:35:59.074 UTC [22449] LOG: shutting down
2026-09-26 12:35:59.280 UTC [22445] LOG: database system is shut down
2026-09-26T12:35:59Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48662 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:35:59 dist-test-slave-t7tk krb5kdc[23947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:35:59 dist-test-slave-t7tk krb5kdc[23947](info): set up 2 sockets
Sep 26 12:35:59 dist-test-slave-t7tk krb5kdc[23947](info): commencing operation
krb5kdc: starting...
W20260926 12:36:00.354486 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.960s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:36:00 dist-test-slave-t7tk krb5kdc[23947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:36:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:36:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:36:00.485459 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:38613
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:38613,127.10.98.61:44853,127.10.98.60:36083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:36:00.598232 23960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:00.598546 23960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:00.598603 23960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:00.602758 23960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:36:00.602833 23960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:00.602859 23960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:00.602881 23960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:36:00.602902 23960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:36:00.608552 23960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:38613,127.10.98.61:44853,127.10.98.60:36083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:38613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:00.609768 23960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:00.610877 23960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:00.619802 23968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:00.620046 23965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:00.619870 23960 server_base.cc:1034] running on GCE node
W20260926 12:36:00.619844 23966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:00.620679 23960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:00.621965 23960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:00.623153 23960 hybrid_clock.cc:648] HybridClock initialized: now 1790426160623107 us; error 61 us; skew 500 ppm
Sep 26 12:36:00 dist-test-slave-t7tk krb5kdc[23947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426160, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:00.626348 23960 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:36:00.627378 23960 webserver.cc:460] Webserver started at http://127.10.98.62:44079/ using document root <none> and password file <none>
I20260926 12:36:00.627947 23960 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:00.628017 23960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:00.628221 23960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:36:00.630405 23960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "0ffc7ece160a4a9f9ee0917031190f97"
format_stamp: "Formatted at 2026-09-26 12:36:00 on dist-test-slave-t7tk"
server_key: "2436c02ca1f66c5ee00c3f6bdfae424d"
server_key_iv: "9867deebda80d87839e0b336b30a24b4"
server_key_version: "encryptionkey@0"
I20260926 12:36:00.630988 23960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "0ffc7ece160a4a9f9ee0917031190f97"
format_stamp: "Formatted at 2026-09-26 12:36:00 on dist-test-slave-t7tk"
server_key: "2436c02ca1f66c5ee00c3f6bdfae424d"
server_key_iv: "9867deebda80d87839e0b336b30a24b4"
server_key_version: "encryptionkey@0"
I20260926 12:36:00.635054 23960 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:36:00.637751 23975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:00.639170 23960 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:36:00.639307 23960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "0ffc7ece160a4a9f9ee0917031190f97"
format_stamp: "Formatted at 2026-09-26 12:36:00 on dist-test-slave-t7tk"
server_key: "2436c02ca1f66c5ee00c3f6bdfae424d"
server_key_iv: "9867deebda80d87839e0b336b30a24b4"
server_key_version: "encryptionkey@0"
I20260926 12:36:00.639420 23960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:00.673380 23960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:00.676626 23960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:00.676816 23960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:00.683660 23960 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:38613
I20260926 12:36:00.683708 24027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:38613 every 8 connection(s)
I20260926 12:36:00.684644 23960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:36:00.687633 24028 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:36:00.691679 24028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:36:00.692615 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 23960
I20260926 12:36:00.692734 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:36:00.693020 10632 external_mini_cluster.cc:1428] Setting key 0e1cea068bdc4674ca261541f5846867
WARNING: no policy specified for kudu/127.10.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.61@KRBTEST.COM" created.
Sep 26 12:36:00 dist-test-slave-t7tk krb5kdc[23947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426160, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:36:00.711038 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.694478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41286 (local address 127.10.98.62:38613)
0926 12:36:00.695052 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:00.695070 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:00.695231 (+ 161us) server_negotiation.cc:407] Connection header received
0926 12:36:00.696385 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:00.696390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:00.696606 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:00.696764 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:00.697830 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:00.698741 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:00.700207 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:00.700582 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:00.700700 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:00.705943 (+ 5243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:00.705951 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:00.705963 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:00.705995 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:00.707879 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:00.708355 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:00.708361 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:00.708363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:00.708476 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:00.708754 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:00.708757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:00.708759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:00.709225 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:00.709348 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:00.709708 (+ 360us) server_negotiation.cc:299] Negotiation successful
0926 12:36:00.709957 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":449,"spinlock_wait_cycles":43392,"thread_start_us":153,"threads_started":1}
I20260926 12:36:00.711045 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.694478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38613 (local address 127.0.0.1:41286)
0926 12:36:00.695050 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:00.695109 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:00.696074 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:00.696811 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:00.696817 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:00.697299 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:00.697659 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:00.697680 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:00.698876 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:00.698879 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:00.700075 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:00.700088 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:00.700746 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:00.700751 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:00.700891 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:00.701516 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:00.701537 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:00.705767 (+ 4230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:00.707997 (+ 2230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:00.708005 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:00.708017 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:00.708250 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:00.708561 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:00.708564 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:00.708566 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:00.708675 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:00.709328 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:00.709334 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:00.709475 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 12:36:00.709636 (+ 161us) client_negotiation.cc:240] Negotiation successful
0926 12:36:00.710022 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"mutex_wait_us":101,"thread_start_us":153,"threads_started":1}
I20260926 12:36:00.713811 24028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:36:00.714565 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.714283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:44853 (local address 127.0.0.1:33500)
0926 12:36:00.714402 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:00.714497 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.61:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 12:36:00.714768 23979 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.98.61:44853: connect: Connection refused (error 111)
I20260926 12:36:00.715788 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.715560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:44853 (local address 127.0.0.1:33510)
0926 12:36:00.715683 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:00.715727 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.61:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 12:36:00.715943 24028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.61:44853: Network error: Client connection negotiation failed: client connection to 127.10.98.61:44853: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.61.keytab.
Entry for principal kudu/127.10.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.61.keytab.
I20260926 12:36:00.725020 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.61:44853
--webserver_interface=127.10.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.61.keytab
--principal=kudu/127.10.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:38613,127.10.98.61:44853,127.10.98.60:36083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:36:00.765193 24028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } attempt: 1
I20260926 12:36:00.766094 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.765759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:44853 (local address 127.0.0.1:33516)
0926 12:36:00.765925 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:00.765997 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.61:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 12:36:00.766896 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.766669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:44853 (local address 127.0.0.1:33522)
0926 12:36:00.766774 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:00.766825 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.61:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 12:36:00.767058 24028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.61:44853: Network error: Client connection negotiation failed: client connection to 127.10.98.61:44853: connect: Connection refused (error 111)
I20260926 12:36:00.835223 24028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } attempt: 2
I20260926 12:36:00.836104 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.835776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:44853 (local address 127.0.0.1:33530)
0926 12:36:00.835934 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:00.836004 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.61:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 12:36:00.836882 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.836667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:44853 (local address 127.0.0.1:33540)
0926 12:36:00.836768 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:00.836818 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.61:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 12:36:00.837047 24028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.61:44853: Network error: Client connection negotiation failed: client connection to 127.10.98.61:44853: connect: Connection refused (error 111)
W20260926 12:36:00.840101 24035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:00.840375 24035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:00.840422 24035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:00.844683 24035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:36:00.844748 24035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:00.844767 24035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:00.844784 24035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:36:00.844802 24035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:36:00.850415 24035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:38613,127.10.98.61:44853,127.10.98.60:36083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.61.keytab
--principal=kudu/127.10.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.61:44853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:00.851648 24035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:00.852723 24035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:00.861509 24041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:00.861509 24040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:00.861598 24035 server_base.cc:1034] running on GCE node
W20260926 12:36:00.861509 24043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:00.862282 24035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:00.863484 24035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:00.864682 24035 hybrid_clock.cc:648] HybridClock initialized: now 1790426160864661 us; error 34 us; skew 500 ppm
Sep 26 12:36:00 dist-test-slave-t7tk krb5kdc[23947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426160, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:00.867589 24035 init.cc:377] Logged in from keytab as kudu/127.10.98.61@KRBTEST.COM (short username kudu)
I20260926 12:36:00.868537 24035 webserver.cc:460] Webserver started at http://127.10.98.61:41531/ using document root <none> and password file <none>
I20260926 12:36:00.869135 24035 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:00.869210 24035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:00.869405 24035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:36:00.871513 24035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-1/data/instance:
uuid: "1d4cf3ddb012412faf96a1e3ba6573ce"
format_stamp: "Formatted at 2026-09-26 12:36:00 on dist-test-slave-t7tk"
server_key: "765644041bdc6274c1e686ffb17ebcf2"
server_key_iv: "9743f17f13520676f49cec102ea7ee62"
server_key_version: "encryptionkey@0"
I20260926 12:36:00.872063 24035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-1/wal/instance:
uuid: "1d4cf3ddb012412faf96a1e3ba6573ce"
format_stamp: "Formatted at 2026-09-26 12:36:00 on dist-test-slave-t7tk"
server_key: "765644041bdc6274c1e686ffb17ebcf2"
server_key_iv: "9743f17f13520676f49cec102ea7ee62"
server_key_version: "encryptionkey@0"
I20260926 12:36:00.876103 24035 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:36:00.878934 24050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:00.880292 24035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:36:00.880422 24035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-1/wal
uuid: "1d4cf3ddb012412faf96a1e3ba6573ce"
format_stamp: "Formatted at 2026-09-26 12:36:00 on dist-test-slave-t7tk"
server_key: "765644041bdc6274c1e686ffb17ebcf2"
server_key_iv: "9743f17f13520676f49cec102ea7ee62"
server_key_version: "encryptionkey@0"
I20260926 12:36:00.880553 24035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:00.894239 24035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:00.897696 24035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:00.897893 24035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:00.905164 24035 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.61:44853
I20260926 12:36:00.905215 24102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.61:44853 every 8 connection(s)
I20260926 12:36:00.906207 24035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-1/data/info.pb
I20260926 12:36:00.909185 24103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:36:00.911474 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 24035
I20260926 12:36:00.911655 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-1/wal/instance
I20260926 12:36:00.911971 10632 external_mini_cluster.cc:1428] Setting key 5c7c6e2e31f6485eebccacd59b5496d8
I20260926 12:36:00.913743 24103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.60@KRBTEST.COM" created.
Sep 26 12:36:00 dist-test-slave-t7tk krb5kdc[23947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426160, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.61@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:36:00.928318 24028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } attempt: 3
I20260926 12:36:00.928471 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.915712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41292 (local address 127.10.98.62:38613)
0926 12:36:00.915844 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:00.915847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:00.916948 (+ 1101us) server_negotiation.cc:407] Connection header received
0926 12:36:00.917926 (+ 978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:00.917930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:00.917993 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:00.918136 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:00.919379 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:00.919942 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:00.921199 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:00.921444 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:00.921549 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:00.924628 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:00.924632 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:00.924634 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:00.924663 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:00.926165 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:00.926737 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:00.926741 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:00.926743 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:00.926832 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:00.927185 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:00.927189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:00.927191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:00.927412 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:00.927512 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:00.928098 (+ 586us) server_negotiation.cc:299] Negotiation successful
0926 12:36:00.928239 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:36:00.929502 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.916252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38613 (local address 127.0.0.1:41292)
0926 12:36:00.916786 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:00.916829 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:00.917714 (+ 885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:00.918324 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:00.918332 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:00.918694 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:00.919229 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:00.919245 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:00.920072 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:00.920075 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:00.921051 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:00.921061 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:00.921609 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:00.921613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:00.921779 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:00.922396 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:00.922417 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:00.924414 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:00.926310 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:00.926317 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:00.926335 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:00.926630 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:00.926924 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:00.926927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:00.926930 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:00.927089 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:00.927525 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:00.927533 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:00.927814 (+ 281us) client_negotiation.cc:769] Sending connection context
0926 12:36:00.928023 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 12:36:00.928383 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":152,"threads_started":1}
I20260926 12:36:00.931083 24103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:36:00 dist-test-slave-t7tk krb5kdc[23947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426160, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for kudu/127.10.98.61@KRBTEST.COM
Sep 26 12:36:00 dist-test-slave-t7tk krb5kdc[23947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426160, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.61@KRBTEST.COM for kudu/127.10.98.61@KRBTEST.COM
I20260926 12:36:00.938918 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.928793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:44853 (local address 127.0.0.1:33546)
0926 12:36:00.929006 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:00.929021 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:00.929126 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:00.929931 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:00.929935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:00.930112 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:00.930244 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:00.930253 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:00.931267 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:00.931271 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:00.932516 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:00.932527 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:00.933171 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:00.933176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:00.933286 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:00.933839 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:00.933853 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:00.935206 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:00.937227 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:00.937230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:00.937232 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:00.937494 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:00.937912 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:00.937915 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:00.937917 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:00.938016 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:00.938463 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:00.938466 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:00.938568 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 12:36:00.938668 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 12:36:00.938790 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260926 12:36:00.939121 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.929024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33546 (local address 127.10.98.61:44853)
0926 12:36:00.929476 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:00.929497 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:00.929522 (+ 25us) server_negotiation.cc:407] Connection header received
0926 12:36:00.929566 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:00.929571 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:00.929777 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:00.929927 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:00.930382 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:00.931150 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:00.932631 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:00.933035 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:00.933143 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:00.935331 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:00.935340 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:00.935352 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:00.935372 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:00.937033 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:00.937670 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:00.937681 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:00.937683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:00.937800 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:00.938104 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:00.938107 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:00.938109 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:00.938363 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:00.938496 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:00.938787 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 12:36:00.938936 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":117,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.60.keytab.
Entry for principal kudu/127.10.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.60.keytab.
I20260926 12:36:00.940508 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.931629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:44853 (local address 127.0.0.1:33548)
0926 12:36:00.931780 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:00.931794 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:00.931861 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:00.932276 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:00.932278 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:00.932404 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:00.932501 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:00.932506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:00.933479 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:00.933482 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:00.933993 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:00.934017 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:00.934447 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:00.934451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:00.934527 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:00.935044 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:00.935059 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:00.936476 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:00.938995 (+ 2519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:00.938998 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:00.939000 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:00.939280 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:00.939555 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:00.939559 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:00.939561 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:00.939643 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:00.940005 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:00.940008 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:00.940099 (+ 91us) client_negotiation.cc:769] Sending connection context
0926 12:36:00.940203 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 12:36:00.940340 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 12:36:00.940732 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.60:36083
--webserver_interface=127.10.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.60.keytab
--principal=kudu/127.10.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:38873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:38613,127.10.98.61:44853,127.10.98.60:36083
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:36:00.940620 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.931700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33548 (local address 127.10.98.61:44853)
0926 12:36:00.932055 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:00.932066 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:00.932081 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:36:00.932117 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:00.932120 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:00.932161 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:00.932258 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:00.932604 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:00.933313 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:00.934112 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:00.934333 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:00.934420 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:00.936608 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:00.936613 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:00.936615 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:00.936638 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:00.938889 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:00.939365 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:00.939368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:00.939370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:00.939442 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:00.939731 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:00.939734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:00.939736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:00.939917 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:00.940004 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:00.940270 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 12:36:00.940408 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":286,"thread_start_us":99,"threads_started":1}
I20260926 12:36:00.941309 24028 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:36:00.941458 24103 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:36:00.941996 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.941757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:36083 (local address 127.0.0.1:52842)
0926 12:36:00.941870 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:00.941924 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 12:36:00.942077 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.941843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:36083 (local address 127.0.0.1:52852)
0926 12:36:00.941942 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:00.942021 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 12:36:00.942232 24052 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.98.60:36083: connect: Connection refused (error 111)
I20260926 12:36:00.943001 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.942707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:36083 (local address 127.0.0.1:52856)
0926 12:36:00.942891 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:00.942937 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260926 12:36:00.943184 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.942945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:36083 (local address 127.0.0.1:52860)
0926 12:36:00.943081 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:00.943122 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":52}
W20260926 12:36:00.943209 24028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.60:36083: Network error: Client connection negotiation failed: client connection to 127.10.98.60:36083: connect: Connection refused (error 111)
W20260926 12:36:00.943310 24103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.60:36083: Network error: Client connection negotiation failed: client connection to 127.10.98.60:36083: connect: Connection refused (error 111)
I20260926 12:36:00.974395 24028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } attempt: 1
I20260926 12:36:00.975349 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.974985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:36083 (local address 127.0.0.1:52862)
0926 12:36:00.975186 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:00.975255 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 12:36:00.976120 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.975898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:36083 (local address 127.0.0.1:52864)
0926 12:36:00.976008 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:00.976056 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 12:36:00.976248 24028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.60:36083: Network error: Client connection negotiation failed: client connection to 127.10.98.60:36083: connect: Connection refused (error 111)
I20260926 12:36:00.992597 24103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } attempt: 1
I20260926 12:36:00.993599 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.993266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:36083 (local address 127.0.0.1:52874)
0926 12:36:00.993420 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:00.993503 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 12:36:00.994391 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:00.994148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:36083 (local address 127.0.0.1:52882)
0926 12:36:00.994266 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:00.994323 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 12:36:00.994524 24103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.60:36083: Network error: Client connection negotiation failed: client connection to 127.10.98.60:36083: connect: Connection refused (error 111)
I20260926 12:36:01.051476 24028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } attempt: 2
I20260926 12:36:01.052315 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:01.052014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:36083 (local address 127.0.0.1:52898)
0926 12:36:01.052171 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:01.052233 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 12:36:01.053434 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:01.053055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:36083 (local address 127.0.0.1:52908)
0926 12:36:01.053258 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:01.053361 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260926 12:36:01.053589 24028 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.60:36083: Network error: Client connection negotiation failed: client connection to 127.10.98.60:36083: connect: Connection refused (error 111)
W20260926 12:36:01.057828 24110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:01.058126 24110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:01.058167 24110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:01.062333 24110 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:36:01.062412 24110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:01.062430 24110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:01.062449 24110 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:36:01.062465 24110 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:36:01.062674 24103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } attempt: 2
I20260926 12:36:01.063578 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:01.063231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:36083 (local address 127.0.0.1:52918)
0926 12:36:01.063402 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:01.063482 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 12:36:01.064347 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:01.064112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:36083 (local address 127.0.0.1:52924)
0926 12:36:01.064223 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:01.064277 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:36083: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 12:36:01.064550 24103 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.60:36083: Network error: Client connection negotiation failed: client connection to 127.10.98.60:36083: connect: Connection refused (error 111)
I20260926 12:36:01.068847 24110 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:38873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:38613,127.10.98.61:44853,127.10.98.60:36083
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.60.keytab
--principal=kudu/127.10.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.60:36083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:01.070266 24110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:01.071408 24110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:01.080798 24116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:01.080790 24119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:01.080824 24117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:01.081202 24110 server_base.cc:1034] running on GCE node
I20260926 12:36:01.081859 24110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:01.083351 24110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:01.084574 24110 hybrid_clock.cc:648] HybridClock initialized: now 1790426161084540 us; error 52 us; skew 500 ppm
Sep 26 12:36:01 dist-test-slave-t7tk krb5kdc[23947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426161, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:01.087917 24110 init.cc:377] Logged in from keytab as kudu/127.10.98.60@KRBTEST.COM (short username kudu)
I20260926 12:36:01.088893 24110 webserver.cc:460] Webserver started at http://127.10.98.60:38085/ using document root <none> and password file <none>
I20260926 12:36:01.089512 24110 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:01.089581 24110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:01.089758 24110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:36:01.091899 24110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-2/data/instance:
uuid: "c22ee24016d3445eb6f030b709d2b702"
format_stamp: "Formatted at 2026-09-26 12:36:01 on dist-test-slave-t7tk"
server_key: "dd9101d85e012f79856c557c8c32dbe1"
server_key_iv: "b4618be9c21da8c4939e27ff026c57e5"
server_key_version: "encryptionkey@0"
I20260926 12:36:01.092478 24110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-2/wal/instance:
uuid: "c22ee24016d3445eb6f030b709d2b702"
format_stamp: "Formatted at 2026-09-26 12:36:01 on dist-test-slave-t7tk"
server_key: "dd9101d85e012f79856c557c8c32dbe1"
server_key_iv: "b4618be9c21da8c4939e27ff026c57e5"
server_key_version: "encryptionkey@0"
I20260926 12:36:01.096740 24110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:36:01.099823 24126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:01.101425 24110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:36:01.101557 24110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-2/wal
uuid: "c22ee24016d3445eb6f030b709d2b702"
format_stamp: "Formatted at 2026-09-26 12:36:01 on dist-test-slave-t7tk"
server_key: "dd9101d85e012f79856c557c8c32dbe1"
server_key_iv: "b4618be9c21da8c4939e27ff026c57e5"
server_key_version: "encryptionkey@0"
I20260926 12:36:01.101703 24110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:01.122606 24110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:01.128026 24110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:01.128288 24110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:01.138659 24110 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.60:36083
I20260926 12:36:01.138754 24178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.60:36083 every 8 connection(s)
I20260926 12:36:01.140136 24110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-2/data/info.pb
I20260926 12:36:01.144481 24179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:36:01.148882 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 24110
I20260926 12:36:01.149036 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790425997379116-10632-0/minicluster-data/master-2/wal/instance
I20260926 12:36:01.149385 10632 external_mini_cluster.cc:1428] Setting key f7bb2bf2742b0553af467f56a618f1cb
I20260926 12:36:01.150070 24179 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:36:01.152756 24028 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } attempt: 3
I20260926 12:36:01.155869 24103 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } attempt: 3
Sep 26 12:36:01 dist-test-slave-t7tk krb5kdc[23947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426160, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for kudu/127.10.98.60@KRBTEST.COM
Sep 26 12:36:01 dist-test-slave-t7tk krb5kdc[23947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426161, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.60@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
Sep 26 12:36:01 dist-test-slave-t7tk krb5kdc[23947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.60@KRBTEST.COM
Sep 26 12:36:01 dist-test-slave-t7tk krb5kdc[23947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426160, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.61@KRBTEST.COM for kudu/127.10.98.60@KRBTEST.COM
I20260926 12:36:01.170908 24032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:01.153186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:36083 (local address 127.0.0.1:52942)
0926 12:36:01.153435 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:01.153449 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:01.153538 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:01.156960 (+ 3422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:01.156965 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:01.157232 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:01.157357 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:01.157367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.158949 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.158953 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:01.159704 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:01.159717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.160289 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.160293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:01.160374 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:01.162590 (+ 2216us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:01.162608 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:01.164313 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:01.168871 (+ 4558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:01.168875 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:01.168878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:01.169190 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:01.169491 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:01.169494 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:01.169496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:01.169578 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:01.170401 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:01.170404 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:01.170502 (+ 98us) client_negotiation.cc:769] Sending connection context
0926 12:36:01.170587 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 12:36:01.170741 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176}
I20260926 12:36:01.172024 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:01.152397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41304 (local address 127.10.98.62:38613)
0926 12:36:01.152550 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:01.152555 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:01.153616 (+ 1061us) server_negotiation.cc:407] Connection header received
0926 12:36:01.153645 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:01.153647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:01.153683 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:01.153745 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:01.158100 (+ 4355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.158846 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.161013 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.161462 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.161546 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:01.165650 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:01.165654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:01.165657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:01.165685 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:01.167742 (+ 2057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.168544 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.168548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.168550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.168646 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.169599 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.169603 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.169605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.169942 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:01.170034 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:01.171743 (+ 1709us) server_negotiation.cc:299] Negotiation successful
0926 12:36:01.171869 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:36:01.172711 24185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:01.153405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52942 (local address 127.10.98.60:36083)
0926 12:36:01.156715 (+ 3310us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:01.156723 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:01.156736 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:36:01.156770 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:01.156772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:01.156814 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:01.156908 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:01.157480 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.158317 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.159813 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.160159 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.160565 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:01.164501 (+ 3936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:01.164508 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:01.164526 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:01.164553 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:01.168746 (+ 4193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.169298 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.169300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.169302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.169396 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.169685 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.169688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.169690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.170273 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:01.171191 (+ 918us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:01.171251 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 12:36:01.171674 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":174,"server-negotiator.queue_time_us":3243,"spinlock_wait_cycles":19200,"thread_start_us":142,"threads_started":1}
I20260926 12:36:01.172711 24182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:01.151155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52940 (local address 127.10.98.60:36083)
0926 12:36:01.151680 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:01.151754 (+ 74us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:01.151807 (+ 53us) server_negotiation.cc:407] Connection header received
0926 12:36:01.152757 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:01.152782 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:01.153220 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:01.154471 (+ 1251us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:01.155158 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.156902 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.158558 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.159629 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.160239 (+ 610us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:01.165581 (+ 5342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:01.165586 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:01.165589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:01.165618 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:01.168542 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.169188 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.169194 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.169196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.169345 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.169685 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.169688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.169690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.170273 (+ 583us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:01.171064 (+ 791us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:01.171203 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 12:36:01.171600 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":357,"spinlock_wait_cycles":5888,"thread_start_us":150,"threads_started":1}
I20260926 12:36:01.173185 24183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:01.152346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38613 (local address 127.0.0.1:41304)
0926 12:36:01.152872 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:01.152927 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:01.153204 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:01.153924 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:01.153935 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:01.154430 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:01.157772 (+ 3342us) spinlock_profiling.cc:243] Waited 1.6 ms on lock 0x2311f5c8. stack: 00007fa6eda8ec78 00007fa6eda8f1a9 00007fa6ed6511d7 00000000005984a6 00000000005984f9 000000000059c488 00007fa6eda8b2ab 00007fa6eda8aad0 00007fa6f68620a5 00007fa6f686258c 00007fa6ee50c1fc 00007fa6f20e1b1f 00007fa6f2122832 00007fa6f212368c 00007fa6f21390d3 00007fa6f213c54b
0926 12:36:01.157940 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:01.157948 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.159954 (+ 2006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.159959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:01.160870 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:01.160882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.161698 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.161701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:01.161851 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:01.162768 (+ 917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:01.162789 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:01.165468 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:01.167877 (+ 2409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:01.167888 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:01.167902 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:01.168403 (+ 501us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:01.169296 (+ 893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:01.169299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:01.169301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:01.169469 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:01.170643 (+ 1174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:01.170650 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:01.170819 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 12:36:01.172635 (+ 1816us) client_negotiation.cc:240] Negotiation successful
0926 12:36:01.172988 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":1,"spinlock_wait_cycles":3686272,"thread_start_us":143,"threads_started":1}
I20260926 12:36:01.174949 24179 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:36:01.175695 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:01.156395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:36083 (local address 127.0.0.1:52946)
0926 12:36:01.156542 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:01.156557 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:01.156648 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:01.160933 (+ 4285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:01.160938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:01.161327 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:01.161468 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:01.161477 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.164175 (+ 2698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.164180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:01.164968 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:01.164986 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.166235 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.166240 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:01.166343 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:01.166947 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:01.166965 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:01.168563 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:01.174081 (+ 5518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:01.174085 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:01.174087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:01.174351 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:01.174667 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:01.174671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:01.174673 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:01.174761 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:01.175261 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:01.175264 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:01.175372 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 12:36:01.175453 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 12:36:01.175573 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 12:36:01.176429 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:01.156421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52946 (local address 127.10.98.60:36083)
0926 12:36:01.160688 (+ 4267us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:01.160698 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:01.160709 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:36:01.160743 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:01.160746 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:01.160786 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:01.160877 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:01.161613 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.164016 (+ 2403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.165562 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.166061 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.166167 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:01.168943 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:01.168947 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:01.168950 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:01.168976 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:01.173914 (+ 4938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.174471 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.174474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.174476 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.174574 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.174859 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.174862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.174864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.175147 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:01.176086 (+ 939us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:01.176161 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 12:36:01.176275 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4040,"thread_start_us":107,"threads_started":1}
I20260926 12:36:01.178071 24028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97: Bootstrap starting.
I20260926 12:36:01.179153 24103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce: Bootstrap starting.
I20260926 12:36:01.180682 24028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97: Neither blocks nor log segments found. Creating new log.
Sep 26 12:36:01 dist-test-slave-t7tk krb5kdc[23947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:36:01.181684 24028 log.cc:824] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97: Log is configured to *not* fsync() on all Append() calls
I20260926 12:36:01.181944 24103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce: Neither blocks nor log segments found. Creating new log.
I20260926 12:36:01.182873 24103 log.cc:824] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce: Log is configured to *not* fsync() on all Append() calls
I20260926 12:36:01.184083 24028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97: No bootstrap required, opened a new log
I20260926 12:36:01.186003 24103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce: No bootstrap required, opened a new log
I20260926 12:36:01.186244 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:01.175990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41306 (local address 127.10.98.62:38613)
0926 12:36:01.176102 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:01.176107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:01.176118 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:36:01.176151 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:01.176154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:01.176197 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:01.176274 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:01.177003 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.177779 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.178829 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.179130 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.179243 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:01.182364 (+ 3121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:01.182368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:01.182371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:01.182405 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:01.184052 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.184618 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.184622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.184624 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.184730 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.185197 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.185201 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.185203 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.185435 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:01.185584 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:01.185966 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 12:36:01.186095 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 12:36:01.188486 24028 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc7ece160a4a9f9ee0917031190f97" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } } peers { permanent_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } } peers { permanent_uuid: "c22ee24016d3445eb6f030b709d2b702" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } }
I20260926 12:36:01.188755 24028 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:36:01.188807 24028 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ffc7ece160a4a9f9ee0917031190f97, State: Initialized, Role: FOLLOWER
Sep 26 12:36:01 dist-test-slave-t7tk krb5kdc[23947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426161, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.60@KRBTEST.COM for kudu/127.10.98.61@KRBTEST.COM
I20260926 12:36:01.189373 24028 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc7ece160a4a9f9ee0917031190f97" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } } peers { permanent_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } } peers { permanent_uuid: "c22ee24016d3445eb6f030b709d2b702" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } }
I20260926 12:36:01.190390 24103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc7ece160a4a9f9ee0917031190f97" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } } peers { permanent_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } } peers { permanent_uuid: "c22ee24016d3445eb6f030b709d2b702" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } }
I20260926 12:36:01.190611 24103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:36:01.190657 24103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4cf3ddb012412faf96a1e3ba6573ce, State: Initialized, Role: FOLLOWER
I20260926 12:36:01.190878 24028 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:36:01.191177 24103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc7ece160a4a9f9ee0917031190f97" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } } peers { permanent_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } } peers { permanent_uuid: "c22ee24016d3445eb6f030b709d2b702" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } }
I20260926 12:36:01.192493 24103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:36:01.193740 24183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:01.177352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:44853 (local address 127.0.0.1:33560)
0926 12:36:01.183433 (+ 6081us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:01.183450 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:01.183615 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:01.183989 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:01.183992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:01.184224 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:01.184406 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:01.184414 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.185604 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.185607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:01.186417 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:01.186429 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.187156 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.187159 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:01.187263 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:01.187713 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:01.187728 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:01.189745 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:01.191840 (+ 2095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:01.191843 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:01.191846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:01.192144 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:01.192534 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:01.192537 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:01.192539 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:01.192639 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:01.193165 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:01.193168 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:01.193273 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 12:36:01.193357 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 12:36:01.193570 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5968}
I20260926 12:36:01.194312 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:01.177430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.10.98.61:44853)
0926 12:36:01.177535 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:01.177539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:01.183529 (+ 5990us) server_negotiation.cc:407] Connection header received
0926 12:36:01.183728 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:01.183731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:01.183783 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:01.183835 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:01.184561 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.185453 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.186565 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.186905 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.186999 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:01.189896 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:01.189900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:01.189903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:01.189927 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:01.191712 (+ 1785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.192276 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.192279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.192281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.192365 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.192774 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.192778 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.192779 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.193041 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:01.194027 (+ 986us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:01.194103 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 12:36:01.194212 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:36:01.190968 24188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc7ece160a4a9f9ee0917031190f97" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } } peers { permanent_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } } peers { permanent_uuid: "c22ee24016d3445eb6f030b709d2b702" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } } }
I20260926 12:36:01.194530 24188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:36:01.194840 24189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc7ece160a4a9f9ee0917031190f97" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } } peers { permanent_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } } peers { permanent_uuid: "c22ee24016d3445eb6f030b709d2b702" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } } }
I20260926 12:36:01.194972 24189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 12:36:01 dist-test-slave-t7tk krb5kdc[23947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426160, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.61@KRBTEST.COM
I20260926 12:36:01.196379 24179 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } has no permanent_uuid. Determining permanent_uuid...
W20260926 12:36:01.199409 24200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:36:01.199503 24200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:36:01.196969 23960 master_runner.cc:186] Error getting master registration for 127.10.98.61:44853: OK, instance_id { permanent_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" instance_seqno: 1790426160903086 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 12:36:01.200304 23960 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 12:36:01.200318 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:01.188432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.10.98.61:44853)
0926 12:36:01.188558 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:01.188562 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:01.188576 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:36:01.188643 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:01.188646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:01.188693 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:01.188739 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:01.189973 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.191269 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.192380 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.192740 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.192819 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:01.196620 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:01.196624 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:01.196626 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:01.196651 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:01.198293 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.198787 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.198791 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.198792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.198890 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.199193 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.199197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.199198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.199406 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:01.199591 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:01.200037 (+ 446us) server_negotiation.cc:299] Negotiation successful
0926 12:36:01.200170 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Sep 26 12:36:01 dist-test-slave-t7tk krb5kdc[23947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426161, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.60@KRBTEST.COM for kudu/127.10.98.60@KRBTEST.COM
W20260926 12:36:01.203702 24211 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:36:01.203781 24211 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:36:01.206453 24183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:01.196838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:36083 (local address 127.0.0.1:52948)
0926 12:36:01.196939 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:01.196952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:01.197027 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:01.197390 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:01.197393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:01.197569 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:01.197671 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:01.197679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.199156 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.199160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:01.199925 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:01.199936 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.200903 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.200906 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:01.200996 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:01.201565 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:01.201580 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:01.203231 (+ 1651us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:01.205021 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:01.205023 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:01.205025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:01.205273 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:01.205524 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:01.205526 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:01.205528 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:01.205595 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:01.205974 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:01.205977 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:01.206051 (+ 74us) client_negotiation.cc:769] Sending connection context
0926 12:36:01.206137 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 12:36:01.206271 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 12:36:01.207201 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:01.196985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52948 (local address 127.10.98.60:36083)
0926 12:36:01.197092 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:01.197115 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:01.197127 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:36:01.197166 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:01.197169 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:01.197214 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:01.197287 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:01.197787 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.198554 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.200320 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.200640 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.200728 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:01.203342 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:01.203346 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:01.203348 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:01.203371 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:01.204913 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.205362 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.205365 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.205367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.205447 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.205666 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.205669 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.205671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.205866 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:01.205945 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:01.206799 (+ 854us) server_negotiation.cc:299] Negotiation successful
0926 12:36:01.207069 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:36:01.207473 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:36:01.204017 24035 master_runner.cc:186] Error getting master registration for 127.10.98.60:36083: OK, instance_id { permanent_uuid: "c22ee24016d3445eb6f030b709d2b702" instance_seqno: 1790426161135895 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 12:36:01.210103 24035 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 12:36:01.210518 24179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702: Bootstrap starting.
I20260926 12:36:01.213491 24179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702: Neither blocks nor log segments found. Creating new log.
I20260926 12:36:01.214429 24179 log.cc:824] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702: Log is configured to *not* fsync() on all Append() calls
I20260926 12:36:01.216740 24179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702: No bootstrap required, opened a new log
I20260926 12:36:01.220795 24108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:01.210450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33576 (local address 127.10.98.61:44853)
0926 12:36:01.210568 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:01.210573 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:01.211152 (+ 579us) server_negotiation.cc:407] Connection header received
0926 12:36:01.211339 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:01.211343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:01.211392 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:01.211444 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:01.212450 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.213265 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.214461 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.214811 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.214888 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:01.216931 (+ 2043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:01.216936 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:01.216938 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:01.216967 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:01.218846 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.219370 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.219374 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.219376 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.219466 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.219773 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.219776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.219778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.220016 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:01.220117 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:01.220495 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 12:36:01.220624 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:36:01.230122 24179 raft_consensus.cc:348] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc7ece160a4a9f9ee0917031190f97" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } } peers { permanent_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } } peers { permanent_uuid: "c22ee24016d3445eb6f030b709d2b702" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } }
I20260926 12:36:01.230432 24179 raft_consensus.cc:374] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:36:01.230485 24179 raft_consensus.cc:729] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c22ee24016d3445eb6f030b709d2b702, State: Initialized, Role: FOLLOWER
I20260926 12:36:01.231072 24179 consensus_queue.cc:260] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc7ece160a4a9f9ee0917031190f97" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } } peers { permanent_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } } peers { permanent_uuid: "c22ee24016d3445eb6f030b709d2b702" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } }
I20260926 12:36:01.232566 24179 sys_catalog.cc:564] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:36:01.234395 24031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:01.217681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41310 (local address 127.10.98.62:38613)
0926 12:36:01.224462 (+ 6781us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:01.224467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:01.224755 (+ 288us) server_negotiation.cc:407] Connection header received
0926 12:36:01.224958 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:01.224962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:01.225013 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:01.225064 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:01.226003 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.226809 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.227943 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.228293 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.228367 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:01.230790 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:01.230794 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:01.230797 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:01.230829 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:01.232505 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.233008 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.233012 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.233014 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.233125 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.233412 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.233415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.233417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.233636 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:01.233751 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:01.234105 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 12:36:01.234231 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6668}
I20260926 12:36:01.235198 24221 sys_catalog.cc:455] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc7ece160a4a9f9ee0917031190f97" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } } peers { permanent_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } } peers { permanent_uuid: "c22ee24016d3445eb6f030b709d2b702" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } } }
I20260926 12:36:01.240490 24221 sys_catalog.cc:458] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:36:01.247874 24110 master_runner.cc:263] No leader master found from master c22ee24016d3445eb6f030b709d2b702
I20260926 12:36:01.247974 24110 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 12:36:01.247951 24232 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:36:01.248004 24232 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:36:01.260538 24184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:01.224956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52964 (local address 127.10.98.60:36083)
0926 12:36:01.225081 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:01.225085 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:01.239439 (+ 14354us) server_negotiation.cc:407] Connection header received
0926 12:36:01.239515 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:01.239519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:01.239595 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:01.239688 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:01.240750 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.241516 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.242688 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:01.243154 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:01.243236 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:01.249407 (+ 6171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:01.249412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:01.249415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:01.249443 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:01.258613 (+ 9170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.259254 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.259257 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.259258 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.259348 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:01.259623 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:01.259627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:01.259629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:01.259828 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:01.259946 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:01.260231 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 12:36:01.260357 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260926 12:36:01.380687 24132 tablet.cc:2369] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:36:01.399659 24056 tablet.cc:2369] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:36:01.428694 23981 tablet.cc:2369] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:36:01.454386 24188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:36:01.454597 24188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc7ece160a4a9f9ee0917031190f97" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } } peers { permanent_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } } peers { permanent_uuid: "c22ee24016d3445eb6f030b709d2b702" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } }
I20260926 12:36:01.455610 24188 leader_election.cc:288] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d4cf3ddb012412faf96a1e3ba6573ce (127.10.98.61:44853), c22ee24016d3445eb6f030b709d2b702 (127.10.98.60:36083)
I20260926 12:36:01.456233 24077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ffc7ece160a4a9f9ee0917031190f97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" is_pre_election: true
I20260926 12:36:01.456228 24153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ffc7ece160a4a9f9ee0917031190f97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c22ee24016d3445eb6f030b709d2b702" is_pre_election: true
I20260926 12:36:01.456568 24153 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ffc7ece160a4a9f9ee0917031190f97 in term 0.
I20260926 12:36:01.456588 24077 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ffc7ece160a4a9f9ee0917031190f97 in term 0.
I20260926 12:36:01.457018 23979 leader_election.cc:302] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ffc7ece160a4a9f9ee0917031190f97, 1d4cf3ddb012412faf96a1e3ba6573ce, c22ee24016d3445eb6f030b709d2b702; no voters:
I20260926 12:36:01.457324 24188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:36:01.457403 24188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:36:01.457436 24188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:36:01.458675 24188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc7ece160a4a9f9ee0917031190f97" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } } peers { permanent_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } } peers { permanent_uuid: "c22ee24016d3445eb6f030b709d2b702" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } }
I20260926 12:36:01.459010 24188 leader_election.cc:288] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [CANDIDATE]: Term 1 election: Requested vote from peers 1d4cf3ddb012412faf96a1e3ba6573ce (127.10.98.61:44853), c22ee24016d3445eb6f030b709d2b702 (127.10.98.60:36083)
I20260926 12:36:01.459412 24077 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ffc7ece160a4a9f9ee0917031190f97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce"
I20260926 12:36:01.459412 24153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ffc7ece160a4a9f9ee0917031190f97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c22ee24016d3445eb6f030b709d2b702"
I20260926 12:36:01.459559 24077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:36:01.459585 24153 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:36:01.460843 24153 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ffc7ece160a4a9f9ee0917031190f97 in term 1.
I20260926 12:36:01.460843 24077 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ffc7ece160a4a9f9ee0917031190f97 in term 1.
I20260926 12:36:01.461203 23979 leader_election.cc:302] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ffc7ece160a4a9f9ee0917031190f97, 1d4cf3ddb012412faf96a1e3ba6573ce; no voters:
I20260926 12:36:01.461369 24188 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:36:01.461644 24188 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [term 1 LEADER]: Becoming Leader. State: Replica: 0ffc7ece160a4a9f9ee0917031190f97, State: Running, Role: LEADER
I20260926 12:36:01.461997 24188 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc7ece160a4a9f9ee0917031190f97" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } } peers { permanent_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } } peers { permanent_uuid: "c22ee24016d3445eb6f030b709d2b702" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } }
I20260926 12:36:01.463835 24234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ffc7ece160a4a9f9ee0917031190f97. Latest consensus state: current_term: 1 leader_uuid: "0ffc7ece160a4a9f9ee0917031190f97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc7ece160a4a9f9ee0917031190f97" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } } peers { permanent_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } } peers { permanent_uuid: "c22ee24016d3445eb6f030b709d2b702" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } } }
I20260926 12:36:01.463943 24234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [sys.catalog]: This master's current role is: LEADER
I20260926 12:36:01.464264 24236 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:36:01.465080 24236 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:36:01.469918 24077 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce [term 1 FOLLOWER]: Refusing update from remote peer 0ffc7ece160a4a9f9ee0917031190f97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:36:01.469916 24153 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702 [term 1 FOLLOWER]: Refusing update from remote peer 0ffc7ece160a4a9f9ee0917031190f97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:36:01.470566 24234 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c22ee24016d3445eb6f030b709d2b702" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:36:01.470734 24188 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:36:01.474465 24221 sys_catalog.cc:455] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ffc7ece160a4a9f9ee0917031190f97. Latest consensus state: current_term: 1 leader_uuid: "0ffc7ece160a4a9f9ee0917031190f97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc7ece160a4a9f9ee0917031190f97" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } } peers { permanent_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } } peers { permanent_uuid: "c22ee24016d3445eb6f030b709d2b702" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } } }
I20260926 12:36:01.474606 24221 sys_catalog.cc:458] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:36:01.476631 24234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ffc7ece160a4a9f9ee0917031190f97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc7ece160a4a9f9ee0917031190f97" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } } peers { permanent_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } } peers { permanent_uuid: "c22ee24016d3445eb6f030b709d2b702" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } } }
I20260926 12:36:01.476763 24234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [sys.catalog]: This master's current role is: LEADER
I20260926 12:36:01.477406 24221 sys_catalog.cc:455] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ffc7ece160a4a9f9ee0917031190f97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc7ece160a4a9f9ee0917031190f97" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } } peers { permanent_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } } peers { permanent_uuid: "c22ee24016d3445eb6f030b709d2b702" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } } }
I20260926 12:36:01.477530 24221 sys_catalog.cc:458] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:36:01.477758 24189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ffc7ece160a4a9f9ee0917031190f97. Latest consensus state: current_term: 1 leader_uuid: "0ffc7ece160a4a9f9ee0917031190f97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc7ece160a4a9f9ee0917031190f97" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } } peers { permanent_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } } peers { permanent_uuid: "c22ee24016d3445eb6f030b709d2b702" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } } }
I20260926 12:36:01.477890 24189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:36:01.477946 24238 mvcc.cc:205] Tried to move back new op lower bound from 7333585557374468096 to 7333585557350981632. Current Snapshot: MvccSnapshot[applied={T|T < 7333585557374468096}]
I20260926 12:36:01.478021 24237 mvcc.cc:205] Tried to move back new op lower bound from 7333585557374468096 to 7333585557350981632. Current Snapshot: MvccSnapshot[applied={T|T < 7333585557374468096}]
I20260926 12:36:01.478699 24236 catalog_manager.cc:1347] Generated new cluster ID: 594ad17cb5d84e0ea69eb890f7da858a
I20260926 12:36:01.478760 24236 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:36:01.479236 24188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0ffc7ece160a4a9f9ee0917031190f97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc7ece160a4a9f9ee0917031190f97" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } } peers { permanent_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } } peers { permanent_uuid: "c22ee24016d3445eb6f030b709d2b702" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } } }
I20260926 12:36:01.479342 24188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97 [sys.catalog]: This master's current role is: LEADER
I20260926 12:36:01.480468 24189 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0ffc7ece160a4a9f9ee0917031190f97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ffc7ece160a4a9f9ee0917031190f97" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38613 } } peers { permanent_uuid: "1d4cf3ddb012412faf96a1e3ba6573ce" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 44853 } } peers { permanent_uuid: "c22ee24016d3445eb6f030b709d2b702" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 36083 } } }
I20260926 12:36:01.480567 24189 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:36:01.487473 24236 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:36:01.488577 24236 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:36:01.495895 24236 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ffc7ece160a4a9f9ee0917031190f97: Generated new TSK 0
I20260926 12:36:01.496897 24236 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:36:02.205087 24211 catalog_manager.cc:1259] Loaded cluster ID: 594ad17cb5d84e0ea69eb890f7da858a
I20260926 12:36:02.205236 24211 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce: loading cluster ID for follower catalog manager: success
I20260926 12:36:02.207387 24211 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce: acquiring CA information for follower catalog manager: success
I20260926 12:36:02.208745 24211 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1d4cf3ddb012412faf96a1e3ba6573ce: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 12:36:02.249531 24232 catalog_manager.cc:1259] Loaded cluster ID: 594ad17cb5d84e0ea69eb890f7da858a
I20260926 12:36:02.249630 24232 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702: loading cluster ID for follower catalog manager: success
I20260926 12:36:02.251271 24232 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702: acquiring CA information for follower catalog manager: success
I20260926 12:36:02.252446 24232 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c22ee24016d3445eb6f030b709d2b702: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 12:36:02.565083 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 23960
I20260926 12:36:02.570659 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 24035
I20260926 12:36:02.576059 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 24110
2026-09-26T12:36:02Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3202 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:36:02 dist-test-slave-t7tk krb5kdc[24252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:36:02 dist-test-slave-t7tk krb5kdc[24252](info): set up 2 sockets
Sep 26 12:36:02 dist-test-slave-t7tk krb5kdc[24252](info): commencing operation
krb5kdc: starting...
W20260926 12:36:03.542728 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:36:03 dist-test-slave-t7tk krb5kdc[24252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:36:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:36:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:36:03.681005 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:41669
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:33387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:41669
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:36:03.792892 24265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:03.793395 24265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:03.793480 24265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:03.798494 24265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:36:03.798563 24265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:03.798581 24265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:03.798596 24265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:36:03.798610 24265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:36:03.804193 24265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:33387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:41669
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:41669
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:03.805342 24265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:03.806454 24265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:03.815354 24270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:03.815536 24265 server_base.cc:1034] running on GCE node
W20260926 12:36:03.815366 24273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:03.815354 24271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:03.816232 24265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:03.817430 24265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:03.818595 24265 hybrid_clock.cc:648] HybridClock initialized: now 1790426163818560 us; error 44 us; skew 500 ppm
Sep 26 12:36:03 dist-test-slave-t7tk krb5kdc[24252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426163, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:03.821555 24265 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:36:03.822482 24265 webserver.cc:460] Webserver started at http://127.10.98.62:32909/ using document root <none> and password file <none>
I20260926 12:36:03.823055 24265 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:03.823130 24265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:03.823323 24265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:36:03.825454 24265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "138f1bc1b9ca4068961bc79eabc479e5"
format_stamp: "Formatted at 2026-09-26 12:36:03 on dist-test-slave-t7tk"
server_key: "4f0f9d1a255be9130b811f00aa266c4c"
server_key_iv: "4f85b475ec26576473d2781bcde382a0"
server_key_version: "encryptionkey@0"
I20260926 12:36:03.825992 24265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "138f1bc1b9ca4068961bc79eabc479e5"
format_stamp: "Formatted at 2026-09-26 12:36:03 on dist-test-slave-t7tk"
server_key: "4f0f9d1a255be9130b811f00aa266c4c"
server_key_iv: "4f85b475ec26576473d2781bcde382a0"
server_key_version: "encryptionkey@0"
I20260926 12:36:03.829751 24265 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:36:03.832439 24280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:03.833858 24265 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:36:03.833983 24265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "138f1bc1b9ca4068961bc79eabc479e5"
format_stamp: "Formatted at 2026-09-26 12:36:03 on dist-test-slave-t7tk"
server_key: "4f0f9d1a255be9130b811f00aa266c4c"
server_key_iv: "4f85b475ec26576473d2781bcde382a0"
server_key_version: "encryptionkey@0"
I20260926 12:36:03.834151 24265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:03.861207 24265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:03.864485 24265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:03.864676 24265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:03.871858 24265 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:41669
I20260926 12:36:03.871929 24332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:41669 every 8 connection(s)
I20260926 12:36:03.872869 24265 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:36:03.875833 24333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:36:03.878119 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 24265
I20260926 12:36:03.878240 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:36:03.878506 10632 external_mini_cluster.cc:1428] Setting key 6525b7300f71c33921ab352a800c4666
I20260926 12:36:03.882333 24333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5: Bootstrap starting.
I20260926 12:36:03.885057 24333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5: Neither blocks nor log segments found. Creating new log.
I20260926 12:36:03.885946 24333 log.cc:824] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:36:03 dist-test-slave-t7tk krb5kdc[24252](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426163, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:36:03.888613 24333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5: No bootstrap required, opened a new log
I20260926 12:36:03.891978 24333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138f1bc1b9ca4068961bc79eabc479e5" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41669 } }
I20260926 12:36:03.892282 24333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:36:03.892346 24333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138f1bc1b9ca4068961bc79eabc479e5, State: Initialized, Role: FOLLOWER
I20260926 12:36:03.892863 24333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138f1bc1b9ca4068961bc79eabc479e5" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41669 } }
I20260926 12:36:03.893014 24333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:36:03.893083 24333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:36:03.893215 24333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:36:03.894312 24333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138f1bc1b9ca4068961bc79eabc479e5" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41669 } }
I20260926 12:36:03.894662 24333 leader_election.cc:302] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 138f1bc1b9ca4068961bc79eabc479e5; no voters:
I20260926 12:36:03.894820 24336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:03.880015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49964 (local address 127.10.98.62:41669)
0926 12:36:03.880416 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:03.880432 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:03.880469 (+ 37us) server_negotiation.cc:407] Connection header received
0926 12:36:03.881289 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:03.881310 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:03.881665 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:03.881951 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:03.882999 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:03.884055 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:03.885349 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:03.885903 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:03.886033 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:03.889222 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:03.889227 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:03.889241 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:03.889270 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:03.891584 (+ 2314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:03.892177 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:03.892182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:03.892187 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:03.892321 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:03.892606 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:03.892608 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:03.892610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:03.893174 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:03.893372 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:03.893633 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 12:36:03.893925 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":102,"threads_started":1}
I20260926 12:36:03.894964 24333 leader_election.cc:288] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:36:03.895227 24338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:36:03.895440 24338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5 [term 1 LEADER]: Becoming Leader. State: Replica: 138f1bc1b9ca4068961bc79eabc479e5, State: Running, Role: LEADER
I20260926 12:36:03.895807 24338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138f1bc1b9ca4068961bc79eabc479e5" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41669 } }
I20260926 12:36:03.896474 24333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:36:03.897513 24340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "138f1bc1b9ca4068961bc79eabc479e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138f1bc1b9ca4068961bc79eabc479e5" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41669 } } }
I20260926 12:36:03.898171 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:36:03.898289 24340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5 [sys.catalog]: This master's current role is: LEADER
I20260926 12:36:03.898181 24339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 138f1bc1b9ca4068961bc79eabc479e5. Latest consensus state: current_term: 1 leader_uuid: "138f1bc1b9ca4068961bc79eabc479e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "138f1bc1b9ca4068961bc79eabc479e5" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41669 } } }
I20260926 12:36:03.898491 24339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5 [sys.catalog]: This master's current role is: LEADER
W20260926 12:36:03.901500 24353 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:36:03.901572 24353 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:36:03.901649 24352 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:36:03.902324 24352 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:36:03.908334 24352 catalog_manager.cc:1347] Generated new cluster ID: 11dd92a0a795411e8bd2bf8f59e7a944
I20260926 12:36:03.908418 24352 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:36:03.917822 24352 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:36:03.919179 24352 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:36:03.934608 24352 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 138f1bc1b9ca4068961bc79eabc479e5: Generated new TSK 0
I20260926 12:36:03.935585 24352 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:36:03.954881 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 24265
2026-09-26T12:36:03Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1377 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:36:04 dist-test-slave-t7tk krb5kdc[24360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:36:04 dist-test-slave-t7tk krb5kdc[24360](info): set up 2 sockets
Sep 26 12:36:04 dist-test-slave-t7tk krb5kdc[24360](info): commencing operation
krb5kdc: starting...
W20260926 12:36:04.918710 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.945s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:36:05 dist-test-slave-t7tk krb5kdc[24360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:36:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:36:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:36:05.053663 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:38593
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:43409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:38593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:36:05.167541 24373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:05.167866 24373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:05.167909 24373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:05.172253 24373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:36:05.172339 24373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:05.172359 24373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:05.172374 24373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:36:05.172389 24373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:36:05.178928 24373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:43409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:38593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:38593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:05.180171 24373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:05.181313 24373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:05.190513 24381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:05.190631 24373 server_base.cc:1034] running on GCE node
W20260926 12:36:05.190582 24379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:05.190873 24378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:05.191632 24373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:05.192885 24373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:05.194139 24373 hybrid_clock.cc:648] HybridClock initialized: now 1790426165194130 us; error 61 us; skew 500 ppm
Sep 26 12:36:05 dist-test-slave-t7tk krb5kdc[24360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426165, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:05.204558 24373 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:36:05.205691 24373 webserver.cc:460] Webserver started at http://127.10.98.62:44687/ using document root <none> and password file <none>
I20260926 12:36:05.206269 24373 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:05.206331 24373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:05.206508 24373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:36:05.208680 24373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "50447deb385645bc844b65f2b1b82e50"
format_stamp: "Formatted at 2026-09-26 12:36:05 on dist-test-slave-t7tk"
server_key: "f9ec68586b7e22de42d79a4c826dcf90"
server_key_iv: "e53b8c9686f716fb28c6bae5c6f3630f"
server_key_version: "encryptionkey@0"
I20260926 12:36:05.209304 24373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "50447deb385645bc844b65f2b1b82e50"
format_stamp: "Formatted at 2026-09-26 12:36:05 on dist-test-slave-t7tk"
server_key: "f9ec68586b7e22de42d79a4c826dcf90"
server_key_iv: "e53b8c9686f716fb28c6bae5c6f3630f"
server_key_version: "encryptionkey@0"
I20260926 12:36:05.213418 24373 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:36:05.216159 24388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:05.217635 24373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:36:05.217756 24373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "50447deb385645bc844b65f2b1b82e50"
format_stamp: "Formatted at 2026-09-26 12:36:05 on dist-test-slave-t7tk"
server_key: "f9ec68586b7e22de42d79a4c826dcf90"
server_key_iv: "e53b8c9686f716fb28c6bae5c6f3630f"
server_key_version: "encryptionkey@0"
I20260926 12:36:05.217857 24373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:05.245540 24373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:05.248960 24373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:05.249200 24373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:05.256014 24373 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:38593
I20260926 12:36:05.256295 24440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:38593 every 8 connection(s)
I20260926 12:36:05.257150 24373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:36:05.259991 24441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:36:05.260926 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 24373
I20260926 12:36:05.261045 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:36:05.261394 10632 external_mini_cluster.cc:1428] Setting key d3c64272415408f468fdb066a847e5ba
I20260926 12:36:05.266860 24441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50: Bootstrap starting.
I20260926 12:36:05.269461 24441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50: Neither blocks nor log segments found. Creating new log.
I20260926 12:36:05.270190 24441 log.cc:824] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:36:05 dist-test-slave-t7tk krb5kdc[24360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426165, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:36:05.272305 24441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50: No bootstrap required, opened a new log
I20260926 12:36:05.276319 24444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:05.262780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54580 (local address 127.10.98.62:38593)
0926 12:36:05.263215 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:05.263233 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:05.263276 (+ 43us) server_negotiation.cc:407] Connection header received
0926 12:36:05.264081 (+ 805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:05.264105 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:05.264520 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:05.264892 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:05.265604 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:05.267050 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:05.267936 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:05.268624 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:05.268776 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:05.271402 (+ 2626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:05.271408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:05.271425 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:05.271450 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:05.273373 (+ 1923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:05.273907 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:05.273911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:05.273916 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:05.274034 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:05.274310 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:05.274312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:05.274313 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:05.274763 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:05.274924 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:05.275225 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 12:36:05.275553 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":111,"threads_started":1}
I20260926 12:36:05.276562 24441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50447deb385645bc844b65f2b1b82e50" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38593 } }
I20260926 12:36:05.276804 24441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:36:05.276862 24441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50447deb385645bc844b65f2b1b82e50, State: Initialized, Role: FOLLOWER
I20260926 12:36:05.277328 24441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50447deb385645bc844b65f2b1b82e50" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38593 } }
I20260926 12:36:05.277473 24441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:36:05.277539 24441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:36:05.277640 24441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:36:05.278568 24441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50447deb385645bc844b65f2b1b82e50" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38593 } }
I20260926 12:36:05.278892 24441 leader_election.cc:302] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50447deb385645bc844b65f2b1b82e50; no voters:
I20260926 12:36:05.279196 24441 leader_election.cc:288] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:36:05.279302 24446 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:36:05.279551 24446 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 1 LEADER]: Becoming Leader. State: Replica: 50447deb385645bc844b65f2b1b82e50, State: Running, Role: LEADER
I20260926 12:36:05.279840 24446 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50447deb385645bc844b65f2b1b82e50" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38593 } }
I20260926 12:36:05.280356 24441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:36:05.281420 24448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50447deb385645bc844b65f2b1b82e50. Latest consensus state: current_term: 1 leader_uuid: "50447deb385645bc844b65f2b1b82e50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50447deb385645bc844b65f2b1b82e50" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38593 } } }
I20260926 12:36:05.281559 24448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [sys.catalog]: This master's current role is: LEADER
I20260926 12:36:05.281838 24447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "50447deb385645bc844b65f2b1b82e50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50447deb385645bc844b65f2b1b82e50" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38593 } } }
I20260926 12:36:05.281956 24447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [sys.catalog]: This master's current role is: LEADER
I20260926 12:36:05.281970 24455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:36:05.285151 24455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:36:05.290660 24455 catalog_manager.cc:1347] Generated new cluster ID: 9d429c5929934c8598e6d4f128415c4b
I20260926 12:36:05.290752 24455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:36:05.303560 24455 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:36:05.304553 24455 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:36:05.311604 24455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50: Generated new TSK 0
I20260926 12:36:05.312503 24455 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
I20260926 12:36:05.353802 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:38593
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:43409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:36:05.467679 24467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:05.467990 24467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:05.468036 24467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:05.472353 24467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:05.472445 24467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:05.472508 24467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:36:05.478385 24467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:43409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:38593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24467
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:05.479645 24467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:05.480782 24467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:05.491253 24473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:05.491266 24472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:05.491454 24467 server_base.cc:1034] running on GCE node
W20260926 12:36:05.491257 24475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:05.492013 24467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:05.492763 24467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:05.493947 24467 hybrid_clock.cc:648] HybridClock initialized: now 1790426165493922 us; error 36 us; skew 500 ppm
Sep 26 12:36:05 dist-test-slave-t7tk krb5kdc[24360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426165, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:05.496870 24467 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:36:05.497857 24467 webserver.cc:460] Webserver started at http://127.10.98.1:36253/ using document root <none> and password file <none>
I20260926 12:36:05.498438 24467 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:05.498502 24467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:05.498711 24467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:36:05.500969 24467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3"
format_stamp: "Formatted at 2026-09-26 12:36:05 on dist-test-slave-t7tk"
server_key: "86689b1600bb9fca1fd09a8fa3575901"
server_key_iv: "c4474883feb4186613bfb253f83b6d51"
server_key_version: "encryptionkey@0"
I20260926 12:36:05.501690 24467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3"
format_stamp: "Formatted at 2026-09-26 12:36:05 on dist-test-slave-t7tk"
server_key: "86689b1600bb9fca1fd09a8fa3575901"
server_key_iv: "c4474883feb4186613bfb253f83b6d51"
server_key_version: "encryptionkey@0"
I20260926 12:36:05.505946 24467 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 12:36:05.509085 24482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:05.510583 24467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:36:05.510716 24467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3"
format_stamp: "Formatted at 2026-09-26 12:36:05 on dist-test-slave-t7tk"
server_key: "86689b1600bb9fca1fd09a8fa3575901"
server_key_iv: "c4474883feb4186613bfb253f83b6d51"
server_key_version: "encryptionkey@0"
I20260926 12:36:05.510829 24467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:05.523257 24467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:05.526618 24467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:05.526784 24467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:05.527354 24467 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:36:05.528178 24467 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:36:05.528226 24467 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:05.528265 24467 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:36:05.528280 24467 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:05.537177 24467 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:34783
I20260926 12:36:05.537297 24595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:34783 every 8 connection(s)
I20260926 12:36:05.538166 24467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:36:05.540675 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 24467
I20260926 12:36:05.540807 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:36:05.541172 10632 external_mini_cluster.cc:1428] Setting key ac42b13c2a91b5e035fab0a5897d732b
Sep 26 12:36:05 dist-test-slave-t7tk krb5kdc[24360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426165, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.2@KRBTEST.COM" created.
I20260926 12:36:05.556722 24444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:05.540161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:60127 (local address 127.10.98.62:38593)
0926 12:36:05.540361 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:05.540366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:05.541513 (+ 1147us) server_negotiation.cc:407] Connection header received
0926 12:36:05.543058 (+ 1545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:05.543062 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:05.543115 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:05.543211 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:05.544911 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:05.545530 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:05.547330 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:05.547630 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:05.547722 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:05.551048 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:05.551052 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:05.551054 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:05.551087 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:05.553165 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:05.553939 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:05.553943 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:05.553946 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:05.554064 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:05.555007 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:05.555011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:05.555013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:05.555295 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:05.555373 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:05.556313 (+ 940us) server_negotiation.cc:299] Negotiation successful
0926 12:36:05.556524 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 12:36:05.558043 24598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:05.540613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.1:60127)
0926 12:36:05.541284 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:05.541369 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:05.542820 (+ 1451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:05.543536 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:05.543545 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:05.543937 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:05.544696 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:05.544719 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:05.545682 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:05.545685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:05.547164 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:05.547180 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:05.547829 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:05.547844 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:05.548083 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:05.548784 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:05.548806 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:05.550736 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:05.553348 (+ 2612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:05.553356 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:05.553378 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:05.553802 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:05.554635 (+ 833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:05.554638 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:05.554641 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:05.554871 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:05.555466 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:05.555473 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:05.555850 (+ 377us) client_negotiation.cc:769] Sending connection context
0926 12:36:05.556092 (+ 242us) client_negotiation.cc:240] Negotiation successful
0926 12:36:05.556662 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":135,"threads_started":1}
I20260926 12:36:05.559674 24596 heartbeater.cc:347] Connected to a master server at 127.10.98.62:38593
I20260926 12:36:05.559957 24596 heartbeater.cc:464] Registering TS with master...
I20260926 12:36:05.560505 24596 heartbeater.cc:511] Master 127.10.98.62:38593 requested a full tablet report, sending...
I20260926 12:36:05.562635 24405 ts_manager.cc:194] Registered new tserver with Master: 1037e58f7ead4eaca6e4ce5ade5fe1c3 (127.10.98.1:34783)
I20260926 12:36:05.564376 24405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:60127
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
I20260926 12:36:05.576658 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:38593
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:43409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:36:05.691819 24601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:05.692142 24601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:05.692207 24601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:05.697062 24601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:05.697217 24601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:05.697292 24601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:36:05.703480 24601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:43409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:38593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:05.704630 24601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:05.705742 24601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:05.716101 24606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:05.716233 24607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:05.716235 24601 server_base.cc:1034] running on GCE node
W20260926 12:36:05.716394 24609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:05.716807 24601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:05.717587 24601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:05.718834 24601 hybrid_clock.cc:648] HybridClock initialized: now 1790426165718803 us; error 47 us; skew 500 ppm
Sep 26 12:36:05 dist-test-slave-t7tk krb5kdc[24360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426165, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:05.721892 24601 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:36:05.722968 24601 webserver.cc:460] Webserver started at http://127.10.98.2:34669/ using document root <none> and password file <none>
I20260926 12:36:05.723557 24601 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:05.723628 24601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:05.723832 24601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:36:05.726115 24601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "166e3a8113ae46b09a9ae5e3c51e52b7"
format_stamp: "Formatted at 2026-09-26 12:36:05 on dist-test-slave-t7tk"
server_key: "bd1b7c29607ed9e6e3dd8a43904baac6"
server_key_iv: "2bdb2f3474af461d917405398806e23e"
server_key_version: "encryptionkey@0"
I20260926 12:36:05.726724 24601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "166e3a8113ae46b09a9ae5e3c51e52b7"
format_stamp: "Formatted at 2026-09-26 12:36:05 on dist-test-slave-t7tk"
server_key: "bd1b7c29607ed9e6e3dd8a43904baac6"
server_key_iv: "2bdb2f3474af461d917405398806e23e"
server_key_version: "encryptionkey@0"
I20260926 12:36:05.730863 24601 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:36:05.733770 24616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:05.735236 24601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:36:05.735374 24601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "166e3a8113ae46b09a9ae5e3c51e52b7"
format_stamp: "Formatted at 2026-09-26 12:36:05 on dist-test-slave-t7tk"
server_key: "bd1b7c29607ed9e6e3dd8a43904baac6"
server_key_iv: "2bdb2f3474af461d917405398806e23e"
server_key_version: "encryptionkey@0"
I20260926 12:36:05.735494 24601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:05.744877 24601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:05.748208 24601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:05.748409 24601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:05.749027 24601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:36:05.749974 24601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:36:05.750022 24601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:05.750087 24601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:36:05.750115 24601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:05.759897 24601 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:35275
I20260926 12:36:05.759936 24729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:35275 every 8 connection(s)
I20260926 12:36:05.760877 24601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:36:05.764873 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 24601
I20260926 12:36:05.764997 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:36:05.765338 10632 external_mini_cluster.cc:1428] Setting key 973156034a54f3ccc9f7a069ba6180ec
Sep 26 12:36:05 dist-test-slave-t7tk krb5kdc[24360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426165, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.3@KRBTEST.COM" created.
I20260926 12:36:05.777906 24444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:05.764773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:50619 (local address 127.10.98.62:38593)
0926 12:36:05.764909 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:05.764914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:05.764944 (+ 30us) server_negotiation.cc:407] Connection header received
0926 12:36:05.766326 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:05.766330 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:05.766384 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:05.766470 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:05.768159 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:05.768978 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:05.770513 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:05.770846 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:05.770962 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:05.774115 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:05.774119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:05.774122 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:05.774155 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:05.775541 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:05.776252 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:05.776255 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:05.776257 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:05.776340 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:05.776707 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:05.776710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:05.776712 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:05.776905 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:05.776996 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:05.777611 (+ 615us) server_negotiation.cc:299] Negotiation successful
0926 12:36:05.777742 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 12:36:05.778924 24732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:05.763501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.2:50619)
0926 12:36:05.764201 (+ 700us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:05.764267 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:05.766032 (+ 1765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:05.766678 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:05.766689 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:05.767238 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:05.767963 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:05.767984 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:05.769177 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:05.769180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:05.770396 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:05.770406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:05.771198 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:05.771206 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:05.771414 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:05.772022 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:05.772039 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:05.773833 (+ 1794us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:05.775679 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:05.775685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:05.775705 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:05.776128 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:05.776435 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:05.776438 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:05.776440 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:05.776602 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:05.777019 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:05.777026 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:05.777356 (+ 330us) client_negotiation.cc:769] Sending connection context
0926 12:36:05.777524 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 12:36:05.777871 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":139,"threads_started":1}
I20260926 12:36:05.780416 24730 heartbeater.cc:347] Connected to a master server at 127.10.98.62:38593
I20260926 12:36:05.780702 24730 heartbeater.cc:464] Registering TS with master...
I20260926 12:36:05.781345 24730 heartbeater.cc:511] Master 127.10.98.62:38593 requested a full tablet report, sending...
I20260926 12:36:05.782577 24405 ts_manager.cc:194] Registered new tserver with Master: 166e3a8113ae46b09a9ae5e3c51e52b7 (127.10.98.2:35275)
I20260926 12:36:05.783200 24405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:50619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
I20260926 12:36:05.793783 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:38593
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:43409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:36:05.907604 24735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:05.907891 24735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:05.907953 24735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:05.912079 24735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:05.912146 24735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:05.912236 24735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:36:05.917743 24735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:43409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:38593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:05.918879 24735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:05.920068 24735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:05.930361 24740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:05.930361 24743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:05.930629 24735 server_base.cc:1034] running on GCE node
W20260926 12:36:05.930665 24741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:05.931243 24735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:05.931927 24735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:05.933142 24735 hybrid_clock.cc:648] HybridClock initialized: now 1790426165933109 us; error 41 us; skew 500 ppm
Sep 26 12:36:05 dist-test-slave-t7tk krb5kdc[24360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426165, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:05.936048 24735 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:36:05.936959 24735 webserver.cc:460] Webserver started at http://127.10.98.3:32821/ using document root <none> and password file <none>
I20260926 12:36:05.937587 24735 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:05.937659 24735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:05.937867 24735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:36:05.940094 24735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523"
format_stamp: "Formatted at 2026-09-26 12:36:05 on dist-test-slave-t7tk"
server_key: "06a6a41ce0be753a2c063f47cfdba23e"
server_key_iv: "d72c0f53385185bb7f809071aca2d4ee"
server_key_version: "encryptionkey@0"
I20260926 12:36:05.940644 24735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523"
format_stamp: "Formatted at 2026-09-26 12:36:05 on dist-test-slave-t7tk"
server_key: "06a6a41ce0be753a2c063f47cfdba23e"
server_key_iv: "d72c0f53385185bb7f809071aca2d4ee"
server_key_version: "encryptionkey@0"
I20260926 12:36:05.944439 24735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:36:05.947109 24750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:05.948596 24735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:36:05.948724 24735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523"
format_stamp: "Formatted at 2026-09-26 12:36:05 on dist-test-slave-t7tk"
server_key: "06a6a41ce0be753a2c063f47cfdba23e"
server_key_iv: "d72c0f53385185bb7f809071aca2d4ee"
server_key_version: "encryptionkey@0"
I20260926 12:36:05.948828 24735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:05.964998 24735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:05.968405 24735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:05.968596 24735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:05.969246 24735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:36:05.970146 24735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:36:05.970194 24735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:05.970250 24735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:36:05.970280 24735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:05.978933 24735 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:36863
I20260926 12:36:05.978981 24863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:36863 every 8 connection(s)
I20260926 12:36:05.979899 24735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:36:05.980146 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 24735
I20260926 12:36:05.980260 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:36:05.980595 10632 external_mini_cluster.cc:1428] Setting key 2c8c8e36ca945f10062c156de5f18814
Sep 26 12:36:05 dist-test-slave-t7tk krb5kdc[24360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426165, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:36:05.993945 24444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:05.981918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:50617 (local address 127.10.98.62:38593)
0926 12:36:05.982063 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:05.982067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:05.983060 (+ 993us) server_negotiation.cc:407] Connection header received
0926 12:36:05.984448 (+ 1388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:05.984451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:05.984501 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:05.984581 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:05.985833 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:05.986364 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:05.987564 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:05.987782 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:05.987869 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:05.990293 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:05.990297 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:05.990299 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:05.990327 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:05.991676 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:05.992212 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:05.992215 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:05.992216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:05.992294 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:05.992639 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:05.992642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:05.992643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:05.992867 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:05.992959 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:05.993639 (+ 680us) server_negotiation.cc:299] Negotiation successful
0926 12:36:05.993772 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:36:05.995055 24866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:05.982418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.3:50617)
0926 12:36:05.982919 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:05.982964 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:05.984233 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:05.984742 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:05.984748 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:05.985163 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:05.985683 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:05.985696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:05.986502 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:05.986505 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:05.987431 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:05.987441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:05.987909 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:05.987912 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:05.988059 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:05.988608 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:05.988624 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:05.990017 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:05.991818 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:05.991823 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:05.991836 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:05.992102 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:05.992373 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:05.992375 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:05.992377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:05.992530 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:05.993010 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:05.993018 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:05.993365 (+ 347us) client_negotiation.cc:769] Sending connection context
0926 12:36:05.993566 (+ 201us) client_negotiation.cc:240] Negotiation successful
0926 12:36:05.993918 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":137,"threads_started":1}
I20260926 12:36:05.996265 24864 heartbeater.cc:347] Connected to a master server at 127.10.98.62:38593
I20260926 12:36:05.996501 24864 heartbeater.cc:464] Registering TS with master...
I20260926 12:36:05.997063 24864 heartbeater.cc:511] Master 127.10.98.62:38593 requested a full tablet report, sending...
I20260926 12:36:05.998188 24405 ts_manager.cc:194] Registered new tserver with Master: ae8c52b4ed4c4024b60c95d5d9eb2523 (127.10.98.3:36863)
I20260926 12:36:05.998772 24405 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:50617
I20260926 12:36:06.007261 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:36:06.016896 24444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:06.009269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54586 (local address 127.10.98.62:38593)
0926 12:36:06.009394 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:06.009399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:06.009551 (+ 152us) server_negotiation.cc:407] Connection header received
0926 12:36:06.009675 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:06.009679 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:06.009727 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:06.009793 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:06.010489 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.011013 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.011921 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.012146 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.012286 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:06.013667 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:06.013671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:06.013673 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:06.013701 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:06.015287 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:06.015713 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:06.015716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:06.015717 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:06.015802 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:06.016123 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:06.016125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:06.016126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:06.016351 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:06.016449 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:06.016639 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 12:36:06.016737 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:36:06.020284 24405 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:36:06.023667 24405 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:36:06.038868 24879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:06.033545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:34783 (local address 127.0.0.1:59514)
0926 12:36:06.034220 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:06.034232 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:06.034306 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:06.034891 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:06.034894 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:06.034906 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:36:06.035086 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:06.035093 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.036095 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.036099 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:06.037352 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:06.037361 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.038518 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.038521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:06.038600 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:06.038620 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 12:36:06.038681 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 12:36:06.038732 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":112,"thread_start_us":75,"threads_started":1}
I20260926 12:36:06.039137 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:06.033701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59514 (local address 127.10.98.1:34783)
0926 12:36:06.034099 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:06.034113 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:06.034384 (+ 271us) server_negotiation.cc:407] Connection header received
0926 12:36:06.034454 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:06.034458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:06.034634 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:06.034752 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:36:06.035202 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.035954 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.037472 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.038315 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.038418 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:06.038556 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:06.038733 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 12:36:06.038839 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":87,"threads_started":1}
I20260926 12:36:06.039180 24878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:06.033009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60982 (local address 127.10.98.2:35275)
0926 12:36:06.033922 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:06.033935 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:06.034090 (+ 155us) server_negotiation.cc:407] Connection header received
0926 12:36:06.034308 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:06.034314 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:06.034520 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:06.034642 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:36:06.035142 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.036183 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.037614 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.038175 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.038274 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:06.038432 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:06.038932 (+ 500us) server_negotiation.cc:299] Negotiation successful
0926 12:36:06.039046 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":813,"thread_start_us":137,"threads_started":1}
I20260926 12:36:06.039371 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:06.032864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45774 (local address 127.10.98.3:36863)
0926 12:36:06.033892 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:06.033910 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:06.033932 (+ 22us) server_negotiation.cc:407] Connection header received
0926 12:36:06.034005 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:06.034010 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:06.034237 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:06.034456 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:36:06.034745 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.035807 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.037581 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.038318 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.038418 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:06.038562 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:06.039003 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 12:36:06.039236 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":926,"thread_start_us":149,"threads_started":1}
I20260926 12:36:06.038868 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:06.032819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:35275 (local address 127.0.0.1:60982)
0926 12:36:06.034000 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:06.034014 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:06.034159 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:06.034844 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:06.034846 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:06.034863 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:36:06.035001 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:06.035009 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.036320 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.036324 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:06.037493 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:06.037500 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.038304 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.038308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:06.038426 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:06.038463 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 12:36:06.038569 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 12:36:06.038690 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"mutex_wait_us":3,"thread_start_us":105,"threads_started":1}
I20260926 12:36:06.039669 24876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:06.032657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:36863 (local address 127.0.0.1:45774)
0926 12:36:06.033566 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:06.033624 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:06.033751 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:06.034459 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:06.034466 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:06.034489 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:36:06.034635 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:06.034642 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.035953 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.035957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:06.037449 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:06.037460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.038621 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.038624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:06.038705 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:06.038729 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:36:06.038795 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 12:36:06.038864 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"mutex_wait_us":85,"thread_start_us":98,"threads_started":1}
I20260926 12:36:06.042277 24664 tablet_service.cc:1467] Processing CreateTablet for tablet e071f8e63a6d4324a2309f0e64ccbb09 (DEFAULT_TABLE table=test-table [id=e0e7cfd0162846ffae2c18a638b3bfd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:36:06.042300 24798 tablet_service.cc:1467] Processing CreateTablet for tablet e071f8e63a6d4324a2309f0e64ccbb09 (DEFAULT_TABLE table=test-table [id=e0e7cfd0162846ffae2c18a638b3bfd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:36:06.042300 24530 tablet_service.cc:1467] Processing CreateTablet for tablet e071f8e63a6d4324a2309f0e64ccbb09 (DEFAULT_TABLE table=test-table [id=e0e7cfd0162846ffae2c18a638b3bfd8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:36:06.043291 24664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e071f8e63a6d4324a2309f0e64ccbb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:36:06.043319 24530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e071f8e63a6d4324a2309f0e64ccbb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:36:06.043326 24798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e071f8e63a6d4324a2309f0e64ccbb09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:36:06.050314 24882 tablet_bootstrap.cc:492] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523: Bootstrap starting.
I20260926 12:36:06.051137 24883 tablet_bootstrap.cc:492] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Bootstrap starting.
I20260926 12:36:06.052830 24882 tablet_bootstrap.cc:654] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523: Neither blocks nor log segments found. Creating new log.
I20260926 12:36:06.052901 24883 tablet_bootstrap.cc:654] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Neither blocks nor log segments found. Creating new log.
I20260926 12:36:06.053511 24881 tablet_bootstrap.cc:492] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7: Bootstrap starting.
I20260926 12:36:06.053673 24882 log.cc:824] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523: Log is configured to *not* fsync() on all Append() calls
I20260926 12:36:06.054157 24883 log.cc:824] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Log is configured to *not* fsync() on all Append() calls
I20260926 12:36:06.055869 24881 tablet_bootstrap.cc:654] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7: Neither blocks nor log segments found. Creating new log.
I20260926 12:36:06.055873 24882 tablet_bootstrap.cc:492] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523: No bootstrap required, opened a new log
I20260926 12:36:06.056066 24882 ts_tablet_manager.cc:1397] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:36:06.056087 24883 tablet_bootstrap.cc:492] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: No bootstrap required, opened a new log
I20260926 12:36:06.056245 24883 ts_tablet_manager.cc:1397] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:36:06.056597 24881 log.cc:824] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7: Log is configured to *not* fsync() on all Append() calls
I20260926 12:36:06.058310 24881 tablet_bootstrap.cc:492] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7: No bootstrap required, opened a new log
I20260926 12:36:06.058488 24881 ts_tablet_manager.cc:1397] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:36:06.060032 24882 raft_consensus.cc:348] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:06.060039 24883 raft_consensus.cc:348] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:06.060310 24882 raft_consensus.cc:374] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:36:06.060309 24883 raft_consensus.cc:374] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:36:06.060359 24882 raft_consensus.cc:729] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8c52b4ed4c4024b60c95d5d9eb2523, State: Initialized, Role: FOLLOWER
I20260926 12:36:06.060367 24883 raft_consensus.cc:729] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1037e58f7ead4eaca6e4ce5ade5fe1c3, State: Initialized, Role: FOLLOWER
I20260926 12:36:06.060853 24882 consensus_queue.cc:260] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:06.060842 24883 consensus_queue.cc:260] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:06.061538 24596 heartbeater.cc:503] Master 127.10.98.62:38593 was elected leader, sending a full tablet report...
I20260926 12:36:06.061908 24882 ts_tablet_manager.cc:1428] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260926 12:36:06.061908 24883 ts_tablet_manager.cc:1428] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260926 12:36:06.061980 24881 raft_consensus.cc:348] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:06.062225 24864 heartbeater.cc:503] Master 127.10.98.62:38593 was elected leader, sending a full tablet report...
I20260926 12:36:06.062234 24881 raft_consensus.cc:374] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:36:06.062281 24881 raft_consensus.cc:729] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166e3a8113ae46b09a9ae5e3c51e52b7, State: Initialized, Role: FOLLOWER
I20260926 12:36:06.062731 24881 consensus_queue.cc:260] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:06.063535 24730 heartbeater.cc:503] Master 127.10.98.62:38593 was elected leader, sending a full tablet report...
I20260926 12:36:06.064785 24881 ts_tablet_manager.cc:1428] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
W20260926 12:36:06.230969 24865 tablet.cc:2369] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:36:06.262100 24731 tablet.cc:2369] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:36:06.289505 24597 tablet.cc:2369] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:36:06.390017 24887 raft_consensus.cc:482] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:36:06.390269 24887 raft_consensus.cc:504] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:06.391444 24887 leader_election.cc:288] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae8c52b4ed4c4024b60c95d5d9eb2523 (127.10.98.3:36863), 166e3a8113ae46b09a9ae5e3c51e52b7 (127.10.98.2:35275)
I20260926 12:36:06.395540 24878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:06.391833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:34287 (local address 127.10.98.2:35275)
0926 12:36:06.391979 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:06.391984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:06.392334 (+ 350us) server_negotiation.cc:407] Connection header received
0926 12:36:06.392465 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:06.392467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:06.392508 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:06.392577 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:36:06.392907 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.393469 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.394589 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.394971 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.395070 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:06.395132 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:06.395333 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 12:36:06.395401 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 12:36:06.395576 24891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:06.391746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:36863 (local address 127.10.98.1:47543)
0926 12:36:06.392157 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:06.392169 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:06.392298 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:06.392595 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:06.392597 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:06.392638 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:36:06.392785 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:06.392791 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.393574 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.393577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:06.394452 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:06.394462 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.395087 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.395091 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:06.395243 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:06.395261 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 12:36:06.395333 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 12:36:06.395401 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":62,"spinlock_wait_cycles":32384,"thread_start_us":124,"threads_started":1}
I20260926 12:36:06.395560 24877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:06.391926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:47543 (local address 127.10.98.3:36863)
0926 12:36:06.392035 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:06.392039 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:06.392258 (+ 219us) server_negotiation.cc:407] Connection header received
0926 12:36:06.392451 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:06.392454 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:06.392501 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:06.392560 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:36:06.392904 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.393461 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.394584 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.394968 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.395061 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:06.395125 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:06.395365 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 12:36:06.395438 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:36:06.395584 24890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:06.391746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:35275 (local address 127.10.98.1:34287)
0926 12:36:06.392255 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:06.392267 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:06.392361 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:06.392598 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:06.392601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:06.392642 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:36:06.392791 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:06.392796 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.393574 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.393577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:06.394457 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:06.394467 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.395093 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.395097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:06.395211 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:06.395235 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:36:06.395293 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 12:36:06.395362 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":121,"spinlock_wait_cycles":4608,"thread_start_us":91,"threads_started":1}
I20260926 12:36:06.396270 24684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e071f8e63a6d4324a2309f0e64ccbb09" candidate_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" is_pre_election: true
I20260926 12:36:06.396270 24818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e071f8e63a6d4324a2309f0e64ccbb09" candidate_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" is_pre_election: true
I20260926 12:36:06.396624 24684 raft_consensus.cc:2454] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1037e58f7ead4eaca6e4ce5ade5fe1c3 in term 0.
I20260926 12:36:06.396624 24818 raft_consensus.cc:2454] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1037e58f7ead4eaca6e4ce5ade5fe1c3 in term 0.
I20260926 12:36:06.397187 24483 leader_election.cc:302] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1037e58f7ead4eaca6e4ce5ade5fe1c3, 166e3a8113ae46b09a9ae5e3c51e52b7, ae8c52b4ed4c4024b60c95d5d9eb2523; no voters:
I20260926 12:36:06.397460 24887 raft_consensus.cc:2781] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:36:06.397555 24887 raft_consensus.cc:482] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:36:06.397596 24887 raft_consensus.cc:3037] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:36:06.398653 24887 raft_consensus.cc:504] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:06.398985 24887 leader_election.cc:288] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 1 election: Requested vote from peers ae8c52b4ed4c4024b60c95d5d9eb2523 (127.10.98.3:36863), 166e3a8113ae46b09a9ae5e3c51e52b7 (127.10.98.2:35275)
I20260926 12:36:06.399410 24818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e071f8e63a6d4324a2309f0e64ccbb09" candidate_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523"
I20260926 12:36:06.399410 24684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e071f8e63a6d4324a2309f0e64ccbb09" candidate_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7"
I20260926 12:36:06.399551 24684 raft_consensus.cc:3037] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:36:06.399541 24818 raft_consensus.cc:3037] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:36:06.400797 24818 raft_consensus.cc:2454] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1037e58f7ead4eaca6e4ce5ade5fe1c3 in term 1.
I20260926 12:36:06.400797 24684 raft_consensus.cc:2454] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1037e58f7ead4eaca6e4ce5ade5fe1c3 in term 1.
I20260926 12:36:06.401165 24483 leader_election.cc:302] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1037e58f7ead4eaca6e4ce5ade5fe1c3, ae8c52b4ed4c4024b60c95d5d9eb2523; no voters:
I20260926 12:36:06.401347 24887 raft_consensus.cc:2781] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:36:06.401600 24887 raft_consensus.cc:686] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 LEADER]: Becoming Leader. State: Replica: 1037e58f7ead4eaca6e4ce5ade5fe1c3, State: Running, Role: LEADER
I20260926 12:36:06.401963 24887 consensus_queue.cc:237] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:06.405303 24403 catalog_manager.cc:5495] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1037e58f7ead4eaca6e4ce5ade5fe1c3 (127.10.98.1). New cstate: current_term: 1 leader_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } health_report { overall_health: HEALTHY } } }
I20260926 12:36:06.498498 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:06.495015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.10.98.1:34783)
0926 12:36:06.495160 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:06.495164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:06.495177 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:36:06.495255 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:06.495257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:06.495312 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:06.495402 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:36:06.495666 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.496211 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.497048 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.497305 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.497395 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:06.497689 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:36:06.498120 (+ 431us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:36:06.498211 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:06.498298 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 12:36:06.498352 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:36:06.504690 24818 raft_consensus.cc:1261] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523 [term 1 FOLLOWER]: Refusing update from remote peer 1037e58f7ead4eaca6e4ce5ade5fe1c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:36:06.504689 24684 raft_consensus.cc:1261] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 1 FOLLOWER]: Refusing update from remote peer 1037e58f7ead4eaca6e4ce5ade5fe1c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:36:06.505476 24887 consensus_queue.cc:1035] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:36:06.505676 24892 consensus_queue.cc:1035] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:36:06.515223 24896 mvcc.cc:205] Tried to move back new op lower bound from 7333585577995636736 to 7333585577584971776. Current Snapshot: MvccSnapshot[applied={T|T < 7333585577995636736}]
I20260926 12:36:06.525609 24403 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:36:06.525830 24403 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:36:06.528573 24403 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50: Sending DeleteTablet for 3 replicas of tablet e071f8e63a6d4324a2309f0e64ccbb09
I20260926 12:36:06.529472 24664 tablet_service.cc:1514] Processing DeleteTablet for tablet e071f8e63a6d4324a2309f0e64ccbb09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:36:06 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:60982
I20260926 12:36:06.529521 24530 tablet_service.cc:1514] Processing DeleteTablet for tablet e071f8e63a6d4324a2309f0e64ccbb09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:36:06 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:59514
I20260926 12:36:06.529588 24798 tablet_service.cc:1514] Processing DeleteTablet for tablet e071f8e63a6d4324a2309f0e64ccbb09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:36:06 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:45774
I20260926 12:36:06.530113 24907 tablet_replica.cc:330] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523: stopping tablet replica
I20260926 12:36:06.530260 24905 tablet_replica.cc:330] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: stopping tablet replica
I20260926 12:36:06.530283 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 24467
I20260926 12:36:06.530483 24907 raft_consensus.cc:2229] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:36:06.530829 24907 raft_consensus.cc:2258] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:36:06.531350 24906 tablet_replica.cc:330] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7: stopping tablet replica
I20260926 12:36:06.531620 24906 raft_consensus.cc:2229] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:36:06.531915 24906 raft_consensus.cc:2258] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:36:06.533041 24907 ts_tablet_manager.cc:1905] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:36:06.533211 24906 ts_tablet_manager.cc:1905] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:36:06.535337 24906 ts_tablet_manager.cc:1918] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:36:06.535445 24906 log.cc:1196] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/e071f8e63a6d4324a2309f0e64ccbb09
I20260926 12:36:06.535704 24906 ts_tablet_manager.cc:1939] T e071f8e63a6d4324a2309f0e64ccbb09 P 166e3a8113ae46b09a9ae5e3c51e52b7: Deleting consensus metadata
I20260926 12:36:06.535874 24907 ts_tablet_manager.cc:1918] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:36:06.535979 24907 log.cc:1196] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/e071f8e63a6d4324a2309f0e64ccbb09
I20260926 12:36:06.536276 24907 ts_tablet_manager.cc:1939] T e071f8e63a6d4324a2309f0e64ccbb09 P ae8c52b4ed4c4024b60c95d5d9eb2523: Deleting consensus metadata
I20260926 12:36:06.537240 24392 catalog_manager.cc:4841] TS 166e3a8113ae46b09a9ae5e3c51e52b7 (127.10.98.2:35275): tablet e071f8e63a6d4324a2309f0e64ccbb09 (table test-table [id=e0e7cfd0162846ffae2c18a638b3bfd8]) successfully deleted
I20260926 12:36:06.537585 24389 catalog_manager.cc:4841] TS ae8c52b4ed4c4024b60c95d5d9eb2523 (127.10.98.3:36863): tablet e071f8e63a6d4324a2309f0e64ccbb09 (table test-table [id=e0e7cfd0162846ffae2c18a638b3bfd8]) successfully deleted
W20260926 12:36:06.538029 24389 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.98.1:34783 (error 108)
I20260926 12:36:06.538162 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 24601
I20260926 12:36:06.539659 24875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:06.539325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:34783 (local address 127.0.0.1:59526)
0926 12:36:06.539462 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:06.539566 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.1:34783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 12:36:06.539860 24389 catalog_manager.cc:4584] TS 1037e58f7ead4eaca6e4ce5ade5fe1c3 (127.10.98.1:34783): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e071f8e63a6d4324a2309f0e64ccbb09: Network error: Client connection negotiation failed: client connection to 127.10.98.1:34783: connect: Connection refused (error 111)
I20260926 12:36:06.544461 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 24735
I20260926 12:36:06.551216 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 24373
I20260926 12:36:06.560552 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:38593
--webserver_interface=127.10.98.62
--webserver_port=44687
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:43409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:38593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:36:06.683130 24909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:06.683427 24909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:06.683467 24909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:06.687583 24909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:36:06.687657 24909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:06.687676 24909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:06.687692 24909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:36:06.687705 24909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:36:06.693224 24909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:43409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:38593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:38593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=44687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:06.694351 24909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:06.695410 24909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:06.708835 24915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:06.708827 24918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:06.708894 24909 server_base.cc:1034] running on GCE node
W20260926 12:36:06.708827 24916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:06.709676 24909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:06.710947 24909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:06.712142 24909 hybrid_clock.cc:648] HybridClock initialized: now 1790426166712090 us; error 34 us; skew 500 ppm
Sep 26 12:36:06 dist-test-slave-t7tk krb5kdc[24360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426166, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:06.714969 24909 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:36:06.715961 24909 webserver.cc:460] Webserver started at http://127.10.98.62:44687/ using document root <none> and password file <none>
I20260926 12:36:06.716552 24909 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:06.716635 24909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:06.720882 24909 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:36:06.723233 24925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:06.724591 24909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:36:06.724722 24909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "50447deb385645bc844b65f2b1b82e50"
format_stamp: "Formatted at 2026-09-26 12:36:05 on dist-test-slave-t7tk"
server_key: "f9ec68586b7e22de42d79a4c826dcf90"
server_key_iv: "e53b8c9686f716fb28c6bae5c6f3630f"
server_key_version: "encryptionkey@0"
I20260926 12:36:06.725458 24909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:06.734670 24909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:06.737709 24909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:06.737895 24909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:06.745059 24909 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:38593
I20260926 12:36:06.745605 24977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:38593 every 8 connection(s)
I20260926 12:36:06.746264 24909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:36:06.747936 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 24909
I20260926 12:36:06.748339 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:34783
--local_ip_for_outbound_sockets=127.10.98.1
--tserver_master_addrs=127.10.98.62:38593
--webserver_port=36253
--webserver_interface=127.10.98.1
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:43409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:36:06.751070 24978 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:36:06.752803 24978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50: Bootstrap starting.
I20260926 12:36:06.769892 24978 log.cc:824] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50: Log is configured to *not* fsync() on all Append() calls
I20260926 12:36:06.778951 24978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:36:06.779320 24978 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50: Bootstrap complete.
I20260926 12:36:06.783324 24978 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50447deb385645bc844b65f2b1b82e50" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38593 } }
I20260926 12:36:06.783759 24978 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50447deb385645bc844b65f2b1b82e50, State: Initialized, Role: FOLLOWER
I20260926 12:36:06.784247 24978 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50447deb385645bc844b65f2b1b82e50" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38593 } }
I20260926 12:36:06.784373 24978 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:36:06.784451 24978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:36:06.784547 24978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 1 FOLLOWER]: Advancing to term 2
I20260926 12:36:06.786640 24978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50447deb385645bc844b65f2b1b82e50" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38593 } }
I20260926 12:36:06.787006 24978 leader_election.cc:302] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50447deb385645bc844b65f2b1b82e50; no voters:
I20260926 12:36:06.787317 24978 leader_election.cc:288] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 12:36:06.787463 24982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 12:36:06.787716 24982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 2 LEADER]: Becoming Leader. State: Replica: 50447deb385645bc844b65f2b1b82e50, State: Running, Role: LEADER
I20260926 12:36:06.788031 24982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50447deb385645bc844b65f2b1b82e50" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38593 } }
I20260926 12:36:06.788561 24978 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:36:06.789580 24983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "50447deb385645bc844b65f2b1b82e50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50447deb385645bc844b65f2b1b82e50" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38593 } } }
I20260926 12:36:06.789714 24983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [sys.catalog]: This master's current role is: LEADER
I20260926 12:36:06.789976 24984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50447deb385645bc844b65f2b1b82e50. Latest consensus state: current_term: 2 leader_uuid: "50447deb385645bc844b65f2b1b82e50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50447deb385645bc844b65f2b1b82e50" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38593 } } }
I20260926 12:36:06.790072 24984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [sys.catalog]: This master's current role is: LEADER
I20260926 12:36:06.790169 24991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:36:06.795418 24991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:36:06.796118 24991 catalog_manager.cc:1259] Loaded cluster ID: 9d429c5929934c8598e6d4f128415c4b
I20260926 12:36:06.796180 24991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:36:06.797529 24991 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 12:36:06.801923 24991 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 12:36:06.808507 24991 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:36:06.809469 24991 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50: Loaded TSK: 0
I20260926 12:36:06.810796 24991 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 12:36:06.866387 24980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:06.866678 24980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:06.866741 24980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:06.870903 24980 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:06.870981 24980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:06.871066 24980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:36:06.876549 24980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:43409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:34783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=36253
--tserver_master_addrs=127.10.98.62:38593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24980
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:06.877807 24980 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:06.878895 24980 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:06.889086 25005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:06.889154 25006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:06.889321 24980 server_base.cc:1034] running on GCE node
W20260926 12:36:06.889070 25008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:06.889919 24980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:06.890592 24980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:06.891775 24980 hybrid_clock.cc:648] HybridClock initialized: now 1790426166891750 us; error 39 us; skew 500 ppm
Sep 26 12:36:06 dist-test-slave-t7tk krb5kdc[24360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426166, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:06.894620 24980 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:36:06.895623 24980 webserver.cc:460] Webserver started at http://127.10.98.1:36253/ using document root <none> and password file <none>
I20260926 12:36:06.896247 24980 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:06.896327 24980 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:06.900429 24980 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:36:06.902805 25015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:06.904119 24980 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:36:06.904227 24980 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3"
format_stamp: "Formatted at 2026-09-26 12:36:05 on dist-test-slave-t7tk"
server_key: "86689b1600bb9fca1fd09a8fa3575901"
server_key_iv: "c4474883feb4186613bfb253f83b6d51"
server_key_version: "encryptionkey@0"
I20260926 12:36:06.904920 24980 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:06.933498 24980 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:06.936710 24980 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:06.936888 24980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:06.937568 24980 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:36:06.938724 25022 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 12:36:06.942983 24980 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 12:36:06.943058 24980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260926 12:36:06.943120 24980 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 12:36:06.944916 24980 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 12:36:06.944975 24980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 12:36:06.945364 25022 tablet_bootstrap.cc:492] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Bootstrap starting.
I20260926 12:36:06.954703 24980 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:34783
I20260926 12:36:06.954824 25129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:34783 every 8 connection(s)
I20260926 12:36:06.955753 24980 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:36:06.963842 25022 log.cc:824] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Log is configured to *not* fsync() on all Append() calls
I20260926 12:36:06.965354 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 24980
I20260926 12:36:06.965837 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:35275
--local_ip_for_outbound_sockets=127.10.98.2
--tserver_master_addrs=127.10.98.62:38593
--webserver_port=34669
--webserver_interface=127.10.98.2
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:43409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:36:06 dist-test-slave-t7tk krb5kdc[24360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426166, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:36:06.973309 25022 tablet_bootstrap.cc:492] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:36:06.973781 25022 tablet_bootstrap.cc:492] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Bootstrap complete.
I20260926 12:36:06.974824 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:06.958194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:46667 (local address 127.10.98.62:38593)
0926 12:36:06.958606 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:06.958633 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:06.958944 (+ 311us) server_negotiation.cc:407] Connection header received
0926 12:36:06.960268 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:06.960286 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:06.960748 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:06.961156 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:06.962356 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.963873 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.965538 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.966280 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.966446 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:06.969571 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:06.969576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:06.969590 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:06.969622 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:06.971301 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:06.971844 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:06.971849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:06.971855 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:06.971989 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:06.972332 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:06.972335 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:06.972336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:06.972716 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:06.972841 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:06.973525 (+ 684us) server_negotiation.cc:299] Negotiation successful
0926 12:36:06.973851 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":123,"threads_started":1}
I20260926 12:36:06.974835 25133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:06.958088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.1:46667)
0926 12:36:06.958760 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:06.958813 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:06.959730 (+ 917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:06.961257 (+ 1527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:06.961264 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:06.961664 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:06.962169 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:06.962184 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.964076 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.964081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:06.965380 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:06.965393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.966445 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.966450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:06.966683 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:06.967717 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:06.967736 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:06.969329 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:06.971431 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:06.971437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:06.971451 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:06.971713 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:06.972087 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:06.972090 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:06.972093 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:06.972225 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:06.972837 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:06.972843 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:06.973156 (+ 313us) client_negotiation.cc:769] Sending connection context
0926 12:36:06.973336 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 12:36:06.973709 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":134,"threads_started":1}
I20260926 12:36:06.975167 25022 ts_tablet_manager.cc:1397] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Time spent bootstrapping tablet: real 0.030s user 0.028s sys 0.000s
I20260926 12:36:06.977339 25130 heartbeater.cc:347] Connected to a master server at 127.10.98.62:38593
I20260926 12:36:06.977596 25130 heartbeater.cc:464] Registering TS with master...
I20260926 12:36:06.978096 25130 heartbeater.cc:511] Master 127.10.98.62:38593 requested a full tablet report, sending...
I20260926 12:36:06.978443 25022 raft_consensus.cc:348] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:06.978973 25022 raft_consensus.cc:729] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1037e58f7ead4eaca6e4ce5ade5fe1c3, State: Initialized, Role: FOLLOWER
I20260926 12:36:06.979529 25022 consensus_queue.cc:260] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:06.980584 25022 ts_tablet_manager.cc:1428] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260926 12:36:06.980667 24942 ts_manager.cc:194] Registered new tserver with Master: 1037e58f7ead4eaca6e4ce5ade5fe1c3 (127.10.98.1:34783)
I20260926 12:36:06.984453 24942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:46667
I20260926 12:36:06.987386 25130 heartbeater.cc:503] Master 127.10.98.62:38593 was elected leader, sending a full tablet report...
Sep 26 12:36:06 dist-test-slave-t7tk krb5kdc[24360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426166, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for kudu/127.10.98.1@KRBTEST.COM
I20260926 12:36:06.994859 25137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:06.983585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:34783 (local address 127.0.0.1:59530)
0926 12:36:06.984043 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:06.984104 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:06.984244 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:06.985011 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:06.985017 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:06.985519 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:06.985652 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:06.985659 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.986649 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.986652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:06.987385 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:06.987396 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.988099 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.988102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:06.988231 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:06.988830 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:06.988845 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:06.990708 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:06.992751 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:06.992763 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:06.992769 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:06.993087 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:06.993437 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:06.993440 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:06.993442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:06.993547 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:06.994119 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:06.994126 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:06.994251 (+ 125us) client_negotiation.cc:769] Sending connection context
0926 12:36:06.994369 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 12:36:06.994629 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":145,"threads_started":1}
I20260926 12:36:06.995080 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:06.983679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.10.98.1:34783)
0926 12:36:06.984536 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:06.984547 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:06.984570 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:36:06.984637 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:06.984643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:06.984844 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:06.984976 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:06.985776 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.986522 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.987551 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:06.987997 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:06.988105 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:06.990874 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:06.990883 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:06.990896 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:06.990919 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:06.992624 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:06.993245 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:06.993250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:06.993251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:06.993339 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:06.993661 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:06.993665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:06.993667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:06.993989 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:06.994117 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:06.994488 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 12:36:06.994653 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":78,"threads_started":1}
I20260926 12:36:06.996003 25064 tablet_service.cc:1514] Processing DeleteTablet for tablet e071f8e63a6d4324a2309f0e64ccbb09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:36:06 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:59530
I20260926 12:36:06.996852 25139 tablet_replica.cc:330] stopping tablet replica
I20260926 12:36:06.997192 25139 raft_consensus.cc:2229] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:36:06.997448 25139 raft_consensus.cc:2258] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:36:06.998518 25139 ts_tablet_manager.cc:1905] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:36:07.002388 25139 ts_tablet_manager.cc:1918] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:36:07.002485 25139 log.cc:1196] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/wal/wals/e071f8e63a6d4324a2309f0e64ccbb09
I20260926 12:36:07.002736 25139 ts_tablet_manager.cc:1939] T e071f8e63a6d4324a2309f0e64ccbb09 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Deleting consensus metadata
I20260926 12:36:07.004679 24926 catalog_manager.cc:4841] TS 1037e58f7ead4eaca6e4ce5ade5fe1c3 (127.10.98.1:34783): tablet e071f8e63a6d4324a2309f0e64ccbb09 (table test-table [id=e0e7cfd0162846ffae2c18a638b3bfd8]) successfully deleted
W20260926 12:36:07.085472 25135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:07.085760 25135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:07.085805 25135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:07.090061 25135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:07.090135 25135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:07.090200 25135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:36:07.095871 25135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:43409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:35275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=34669
--tserver_master_addrs=127.10.98.62:38593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:07.097010 25135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:07.098068 25135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:07.108103 25144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:07.108103 25145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:07.108108 25147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:07.108371 25135 server_base.cc:1034] running on GCE node
I20260926 12:36:07.108928 25135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:07.109771 25135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:07.110975 25135 hybrid_clock.cc:648] HybridClock initialized: now 1790426167110956 us; error 41 us; skew 500 ppm
Sep 26 12:36:07 dist-test-slave-t7tk krb5kdc[24360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426167, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:07.114120 25135 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:36:07.115024 25135 webserver.cc:460] Webserver started at http://127.10.98.2:34669/ using document root <none> and password file <none>
I20260926 12:36:07.115579 25135 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:07.115653 25135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:07.119810 25135 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:36:07.122174 25154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:07.123625 25135 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260926 12:36:07.123750 25135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "166e3a8113ae46b09a9ae5e3c51e52b7"
format_stamp: "Formatted at 2026-09-26 12:36:05 on dist-test-slave-t7tk"
server_key: "bd1b7c29607ed9e6e3dd8a43904baac6"
server_key_iv: "2bdb2f3474af461d917405398806e23e"
server_key_version: "encryptionkey@0"
I20260926 12:36:07.124333 25135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:07.147220 25135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:07.150720 25135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:07.150920 25135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:07.151593 25135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:36:07.152459 25135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:36:07.152508 25135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:07.152560 25135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:36:07.152590 25135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:07.161522 25135 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:35275
I20260926 12:36:07.161543 25267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:35275 every 8 connection(s)
I20260926 12:36:07.162585 25135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:36:07.163568 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 25135
I20260926 12:36:07.163988 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:36863
--local_ip_for_outbound_sockets=127.10.98.3
--tserver_master_addrs=127.10.98.62:38593
--webserver_port=32821
--webserver_interface=127.10.98.3
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:43409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:36:07 dist-test-slave-t7tk krb5kdc[24360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426167, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:36:07.177403 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:07.165250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:33703 (local address 127.10.98.62:38593)
0926 12:36:07.165388 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:07.165393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:07.165939 (+ 546us) server_negotiation.cc:407] Connection header received
0926 12:36:07.167680 (+ 1741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:07.167683 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:07.167734 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:07.167829 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:07.169204 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.169744 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.170905 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.171144 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.171237 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:07.173698 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:07.173701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:07.173703 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:07.173731 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:07.175210 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:07.175754 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:07.175757 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:07.175759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:07.175844 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:07.176205 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:07.176207 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:07.176208 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:07.176405 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:07.176490 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:07.177040 (+ 550us) server_negotiation.cc:299] Negotiation successful
0926 12:36:07.177219 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:36:07.178212 25271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:07.165175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.2:33703)
0926 12:36:07.165744 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:07.165800 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:07.167410 (+ 1610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:07.168110 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:07.168118 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:07.168503 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:07.169015 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:07.169029 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.169884 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.169887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:07.170773 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:07.170782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.171297 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.171301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:07.171465 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:07.172027 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:07.172044 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:07.173515 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:07.175343 (+ 1828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:07.175349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:07.175367 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:07.175640 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:07.175938 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:07.175941 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:07.175943 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:07.176102 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:07.176510 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:07.176517 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:07.176782 (+ 265us) client_negotiation.cc:769] Sending connection context
0926 12:36:07.176960 (+ 178us) client_negotiation.cc:240] Negotiation successful
0926 12:36:07.177344 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":154,"threads_started":1}
I20260926 12:36:07.179492 25268 heartbeater.cc:347] Connected to a master server at 127.10.98.62:38593
I20260926 12:36:07.179733 25268 heartbeater.cc:464] Registering TS with master...
I20260926 12:36:07.180296 25268 heartbeater.cc:511] Master 127.10.98.62:38593 requested a full tablet report, sending...
I20260926 12:36:07.181442 24942 ts_manager.cc:194] Registered new tserver with Master: 166e3a8113ae46b09a9ae5e3c51e52b7 (127.10.98.2:35275)
I20260926 12:36:07.182121 24942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:33703
W20260926 12:36:07.279623 25270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:07.279922 25270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:07.279982 25270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:07.284616 25270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:07.284703 25270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:07.284788 25270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:36:07.290477 25270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:43409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:36863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=32821
--tserver_master_addrs=127.10.98.62:38593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:07.291601 25270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:07.292658 25270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:07.302600 25276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:07.302604 25277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:07.302600 25279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:07.302851 25270 server_base.cc:1034] running on GCE node
I20260926 12:36:07.303354 25270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:07.304098 25270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:07.305294 25270 hybrid_clock.cc:648] HybridClock initialized: now 1790426167305252 us; error 57 us; skew 500 ppm
Sep 26 12:36:07 dist-test-slave-t7tk krb5kdc[24360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426167, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:07.308267 25270 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:36:07.309365 25270 webserver.cc:460] Webserver started at http://127.10.98.3:32821/ using document root <none> and password file <none>
I20260926 12:36:07.309940 25270 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:07.310019 25270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:07.313992 25270 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:36:07.316450 25286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:07.317854 25270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:36:07.317970 25270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523"
format_stamp: "Formatted at 2026-09-26 12:36:05 on dist-test-slave-t7tk"
server_key: "06a6a41ce0be753a2c063f47cfdba23e"
server_key_iv: "d72c0f53385185bb7f809071aca2d4ee"
server_key_version: "encryptionkey@0"
I20260926 12:36:07.318588 25270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:07.337332 25270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:07.340549 25270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:07.340735 25270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:07.341394 25270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:36:07.342247 25270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:36:07.342295 25270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:07.342347 25270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:36:07.342375 25270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:07.351112 25270 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:36863
I20260926 12:36:07.351155 25399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:36863 every 8 connection(s)
I20260926 12:36:07.352214 25270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:36:07.361505 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 25270
Sep 26 12:36:07 dist-test-slave-t7tk krb5kdc[24360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426167, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:36:07.366446 25132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:07.354193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:36749 (local address 127.10.98.62:38593)
0926 12:36:07.354358 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:07.354362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:07.355288 (+ 926us) server_negotiation.cc:407] Connection header received
0926 12:36:07.356666 (+ 1378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:07.356669 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:07.356720 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:07.356797 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:07.358118 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.358645 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.359851 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.360082 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.360165 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:07.362967 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:07.362970 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:07.362972 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:07.362996 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:07.364300 (+ 1304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:07.364809 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:07.364811 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:07.364813 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:07.364895 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:07.365259 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:07.365262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:07.365264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:07.365446 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:07.365535 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:07.366158 (+ 623us) server_negotiation.cc:299] Negotiation successful
0926 12:36:07.366279 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 12:36:07.367524 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:07.354604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.3:36749)
0926 12:36:07.355112 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:07.355159 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:07.356424 (+ 1265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:07.356967 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:07.356975 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:07.357459 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:07.357958 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:07.357971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.358771 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.358774 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:07.359723 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:07.359733 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.360245 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.360249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:07.360412 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:07.360966 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:07.360984 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:07.362783 (+ 1799us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:07.364425 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:07.364431 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:07.364449 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:07.364707 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:07.364983 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:07.364986 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:07.364988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:07.365171 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:07.365552 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:07.365559 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:07.365875 (+ 316us) client_negotiation.cc:769] Sending connection context
0926 12:36:07.366082 (+ 207us) client_negotiation.cc:240] Negotiation successful
0926 12:36:07.366439 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":114,"threads_started":1}
I20260926 12:36:07.368794 25400 heartbeater.cc:347] Connected to a master server at 127.10.98.62:38593
I20260926 12:36:07.369035 25400 heartbeater.cc:464] Registering TS with master...
I20260926 12:36:07.369645 25400 heartbeater.cc:511] Master 127.10.98.62:38593 requested a full tablet report, sending...
I20260926 12:36:07.370014 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:07.362338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54588 (local address 127.10.98.62:38593)
0926 12:36:07.362602 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:07.362606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:07.362746 (+ 140us) server_negotiation.cc:407] Connection header received
0926 12:36:07.362867 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:07.362870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:07.362911 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:07.362986 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:07.363697 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.364233 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.365073 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.365331 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.365421 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:07.366765 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:07.366769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:07.366771 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:07.366794 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:07.368395 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:07.368810 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:07.368813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:07.368815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:07.368900 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:07.369184 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:07.369188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:07.369189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:07.369416 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:07.369615 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:07.369736 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 12:36:07.369857 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":66,"threads_started":1}
I20260926 12:36:07.370802 24942 ts_manager.cc:194] Registered new tserver with Master: ae8c52b4ed4c4024b60c95d5d9eb2523 (127.10.98.3:36863)
I20260926 12:36:07.371382 24942 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:36749
I20260926 12:36:07.371773 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:36:07.376181 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:07.372369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54600 (local address 127.10.98.62:38593)
0926 12:36:07.372539 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:07.372543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:07.372683 (+ 140us) server_negotiation.cc:407] Connection header received
0926 12:36:07.372842 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:07.372845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:07.372957 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:07.373046 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:36:07.373313 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.373826 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.374736 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.374970 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.375077 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:07.375401 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:36:07.375776 (+ 375us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:36:07.375865 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:07.375971 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 12:36:07.376026 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260926 12:36:07.376852 24942 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:36:07.380216 24942 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:36:07.388315 25064 tablet_service.cc:1467] Processing CreateTablet for tablet 4f77897b176547f786bc8f8bc7a083a5 (DEFAULT_TABLE table=test-table [id=36345759f03c47979ec09e5813efcc66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:36:07.389055 25064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f77897b176547f786bc8f8bc7a083a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:36:07.392351 25022 tablet_bootstrap.cc:492] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Bootstrap starting.
I20260926 12:36:07.393294 25022 tablet_bootstrap.cc:654] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Neither blocks nor log segments found. Creating new log.
I20260926 12:36:07.393700 25137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:07.387335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:36863 (local address 127.0.0.1:45778)
0926 12:36:07.387476 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:07.387494 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:07.387579 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:07.388473 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:07.388476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:07.388496 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:36:07.388625 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:07.388634 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.390287 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.390289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:07.391584 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:07.391649 (+ 65us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.393158 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.393162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:07.393269 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:07.393331 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 12:36:07.393412 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 12:36:07.393526 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 12:36:07.394006 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:07.388090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:35275 (local address 127.0.0.1:60988)
0926 12:36:07.388363 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:07.388377 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:07.388463 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:07.389912 (+ 1449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:07.389916 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:07.389937 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:36:07.390037 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:07.390043 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.391206 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.391209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:07.392057 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:07.392065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.393480 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.393485 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:07.393592 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:07.393612 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 12:36:07.393685 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 12:36:07.393757 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":103,"threads_started":1}
I20260926 12:36:07.394176 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:07.387516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45778 (local address 127.10.98.3:36863)
0926 12:36:07.387848 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:07.387860 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:07.387885 (+ 25us) server_negotiation.cc:407] Connection header received
0926 12:36:07.387971 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:07.387977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:07.388283 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:07.388832 (+ 549us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:36:07.388977 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.390163 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.391803 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.392922 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.393041 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:07.393238 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:07.393897 (+ 659us) server_negotiation.cc:299] Negotiation successful
0926 12:36:07.394025 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":151,"threads_started":1}
I20260926 12:36:07.395092 25022 tablet_bootstrap.cc:492] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: No bootstrap required, opened a new log
I20260926 12:36:07.395183 25022 ts_tablet_manager.cc:1397] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:36:07.395200 25411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:07.388524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60988 (local address 127.10.98.2:35275)
0926 12:36:07.389381 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:07.389392 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:07.389416 (+ 24us) server_negotiation.cc:407] Connection header received
0926 12:36:07.389493 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:07.389498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:07.389700 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:07.389874 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:36:07.390165 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.391073 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.392187 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.393174 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.393269 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:07.393397 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:07.393742 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 12:36:07.394216 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":100,"threads_started":1}
I20260926 12:36:07.395653 25022 raft_consensus.cc:348] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:07.395782 25022 raft_consensus.cc:374] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:36:07.395817 25022 raft_consensus.cc:729] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1037e58f7ead4eaca6e4ce5ade5fe1c3, State: Initialized, Role: FOLLOWER
I20260926 12:36:07.395941 25022 consensus_queue.cc:260] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:07.396212 25022 ts_tablet_manager.cc:1428] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:36:07.397428 25334 tablet_service.cc:1467] Processing CreateTablet for tablet 4f77897b176547f786bc8f8bc7a083a5 (DEFAULT_TABLE table=test-table [id=36345759f03c47979ec09e5813efcc66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:36:07.397943 25202 tablet_service.cc:1467] Processing CreateTablet for tablet 4f77897b176547f786bc8f8bc7a083a5 (DEFAULT_TABLE table=test-table [id=36345759f03c47979ec09e5813efcc66]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:36:07.398473 25334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f77897b176547f786bc8f8bc7a083a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:36:07.398967 25202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f77897b176547f786bc8f8bc7a083a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:36:07.405429 25413 tablet_bootstrap.cc:492] T 4f77897b176547f786bc8f8bc7a083a5 P ae8c52b4ed4c4024b60c95d5d9eb2523: Bootstrap starting.
I20260926 12:36:07.405424 25414 tablet_bootstrap.cc:492] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7: Bootstrap starting.
I20260926 12:36:07.407791 25413 tablet_bootstrap.cc:654] T 4f77897b176547f786bc8f8bc7a083a5 P ae8c52b4ed4c4024b60c95d5d9eb2523: Neither blocks nor log segments found. Creating new log.
I20260926 12:36:07.407792 25414 tablet_bootstrap.cc:654] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7: Neither blocks nor log segments found. Creating new log.
I20260926 12:36:07.408680 25413 log.cc:824] T 4f77897b176547f786bc8f8bc7a083a5 P ae8c52b4ed4c4024b60c95d5d9eb2523: Log is configured to *not* fsync() on all Append() calls
I20260926 12:36:07.408686 25414 log.cc:824] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7: Log is configured to *not* fsync() on all Append() calls
I20260926 12:36:07.410506 25414 tablet_bootstrap.cc:492] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7: No bootstrap required, opened a new log
I20260926 12:36:07.410516 25413 tablet_bootstrap.cc:492] T 4f77897b176547f786bc8f8bc7a083a5 P ae8c52b4ed4c4024b60c95d5d9eb2523: No bootstrap required, opened a new log
I20260926 12:36:07.410691 25413 ts_tablet_manager.cc:1397] T 4f77897b176547f786bc8f8bc7a083a5 P ae8c52b4ed4c4024b60c95d5d9eb2523: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:36:07.410691 25414 ts_tablet_manager.cc:1397] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:36:07.413862 25413 raft_consensus.cc:348] T 4f77897b176547f786bc8f8bc7a083a5 P ae8c52b4ed4c4024b60c95d5d9eb2523 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:07.413861 25414 raft_consensus.cc:348] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:07.414096 25414 raft_consensus.cc:374] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:36:07.414099 25413 raft_consensus.cc:374] T 4f77897b176547f786bc8f8bc7a083a5 P ae8c52b4ed4c4024b60c95d5d9eb2523 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:36:07.414135 25414 raft_consensus.cc:729] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 166e3a8113ae46b09a9ae5e3c51e52b7, State: Initialized, Role: FOLLOWER
I20260926 12:36:07.414152 25413 raft_consensus.cc:729] T 4f77897b176547f786bc8f8bc7a083a5 P ae8c52b4ed4c4024b60c95d5d9eb2523 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae8c52b4ed4c4024b60c95d5d9eb2523, State: Initialized, Role: FOLLOWER
I20260926 12:36:07.414561 25414 consensus_queue.cc:260] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:07.414561 25413 consensus_queue.cc:260] T 4f77897b176547f786bc8f8bc7a083a5 P ae8c52b4ed4c4024b60c95d5d9eb2523 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:07.415407 25400 heartbeater.cc:503] Master 127.10.98.62:38593 was elected leader, sending a full tablet report...
I20260926 12:36:07.415493 25414 ts_tablet_manager.cc:1428] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260926 12:36:07.415575 25413 ts_tablet_manager.cc:1428] T 4f77897b176547f786bc8f8bc7a083a5 P ae8c52b4ed4c4024b60c95d5d9eb2523: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:36:07.415709 25268 heartbeater.cc:503] Master 127.10.98.62:38593 was elected leader, sending a full tablet report...
W20260926 12:36:07.457070 25131 tablet.cc:2369] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:36:07.464232 25418 raft_consensus.cc:482] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:36:07.464507 25418 raft_consensus.cc:504] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:07.465770 25418 leader_election.cc:288] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae8c52b4ed4c4024b60c95d5d9eb2523 (127.10.98.3:36863), 1037e58f7ead4eaca6e4ce5ade5fe1c3 (127.10.98.1:34783)
I20260926 12:36:07.470309 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:07.466161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:40355 (local address 127.10.98.3:36863)
0926 12:36:07.466310 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:07.466314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:07.466330 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:36:07.466388 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:07.466391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:07.466445 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:07.466527 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:36:07.466901 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.467629 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.469176 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.469762 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.469869 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:07.469939 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:07.470104 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 12:36:07.470182 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 12:36:07.470306 25271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:07.466055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:36863 (local address 127.10.98.2:40355)
0926 12:36:07.466174 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:07.466190 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:07.466286 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:07.466640 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:07.466644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:07.466662 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:36:07.466786 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:07.466793 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.467766 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.467769 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:07.469019 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:07.469030 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.469881 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.469884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:07.469973 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:07.469994 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:36:07.470065 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 12:36:07.470159 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":1}
I20260926 12:36:07.470510 25138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:07.466469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:47053 (local address 127.10.98.1:34783)
0926 12:36:07.466592 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:07.466595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:07.466763 (+ 168us) server_negotiation.cc:407] Connection header received
0926 12:36:07.466832 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:07.466835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:07.466900 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:07.467001 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:36:07.467302 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.467956 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.469330 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.469968 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.470054 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:07.470235 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:07.470325 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 12:36:07.470401 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:36:07.470731 25419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:07.466311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:34783 (local address 127.10.98.2:47053)
0926 12:36:07.466660 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:07.466672 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:07.466758 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:07.467028 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:07.467030 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:07.467101 (+ 71us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:36:07.467193 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:07.467198 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.468077 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.468080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:07.469211 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:07.469219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.470087 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.470090 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:07.470213 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:07.470235 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:36:07.470295 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 12:36:07.470354 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":22,"thread_start_us":131,"threads_started":1}
I20260926 12:36:07.471226 25354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f77897b176547f786bc8f8bc7a083a5" candidate_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" is_pre_election: true
I20260926 12:36:07.471287 25083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f77897b176547f786bc8f8bc7a083a5" candidate_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" is_pre_election: true
I20260926 12:36:07.471580 25083 raft_consensus.cc:2454] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 166e3a8113ae46b09a9ae5e3c51e52b7 in term 0.
I20260926 12:36:07.471577 25354 raft_consensus.cc:2454] T 4f77897b176547f786bc8f8bc7a083a5 P ae8c52b4ed4c4024b60c95d5d9eb2523 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 166e3a8113ae46b09a9ae5e3c51e52b7 in term 0.
I20260926 12:36:07.472081 25155 leader_election.cc:302] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 166e3a8113ae46b09a9ae5e3c51e52b7, ae8c52b4ed4c4024b60c95d5d9eb2523; no voters:
I20260926 12:36:07.472357 25418 raft_consensus.cc:2781] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:36:07.472448 25418 raft_consensus.cc:482] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:36:07.472497 25418 raft_consensus.cc:3037] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:36:07.473654 25418 raft_consensus.cc:504] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:07.474005 25418 leader_election.cc:288] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 [CANDIDATE]: Term 1 election: Requested vote from peers ae8c52b4ed4c4024b60c95d5d9eb2523 (127.10.98.3:36863), 1037e58f7ead4eaca6e4ce5ade5fe1c3 (127.10.98.1:34783)
I20260926 12:36:07.474452 25354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f77897b176547f786bc8f8bc7a083a5" candidate_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523"
I20260926 12:36:07.474524 25083 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f77897b176547f786bc8f8bc7a083a5" candidate_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3"
I20260926 12:36:07.474601 25354 raft_consensus.cc:3037] T 4f77897b176547f786bc8f8bc7a083a5 P ae8c52b4ed4c4024b60c95d5d9eb2523 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:36:07.474661 25083 raft_consensus.cc:3037] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:36:07.475965 25083 raft_consensus.cc:2454] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166e3a8113ae46b09a9ae5e3c51e52b7 in term 1.
I20260926 12:36:07.475965 25354 raft_consensus.cc:2454] T 4f77897b176547f786bc8f8bc7a083a5 P ae8c52b4ed4c4024b60c95d5d9eb2523 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 166e3a8113ae46b09a9ae5e3c51e52b7 in term 1.
I20260926 12:36:07.476295 25155 leader_election.cc:302] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1037e58f7ead4eaca6e4ce5ade5fe1c3, 166e3a8113ae46b09a9ae5e3c51e52b7; no voters:
I20260926 12:36:07.476480 25418 raft_consensus.cc:2781] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:36:07.476722 25418 raft_consensus.cc:686] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 1 LEADER]: Becoming Leader. State: Replica: 166e3a8113ae46b09a9ae5e3c51e52b7, State: Running, Role: LEADER
I20260926 12:36:07.477072 25418 consensus_queue.cc:237] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:07.480391 24938 catalog_manager.cc:5495] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 166e3a8113ae46b09a9ae5e3c51e52b7 (127.10.98.2). New cstate: current_term: 1 leader_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } health_report { overall_health: UNKNOWN } } }
I20260926 12:36:07.511234 25411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:07.507374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60996 (local address 127.10.98.2:35275)
0926 12:36:07.507523 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:07.507528 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:07.507543 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:36:07.507646 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:07.507648 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:07.507708 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:07.507800 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:36:07.508027 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.508718 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.509644 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:07.509865 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:07.510004 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:07.510346 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:36:07.510795 (+ 449us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:36:07.510899 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:07.511041 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 12:36:07.511102 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:36:07.517663 25083 raft_consensus.cc:1261] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Refusing update from remote peer 166e3a8113ae46b09a9ae5e3c51e52b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:36:07.517664 25354 raft_consensus.cc:1261] T 4f77897b176547f786bc8f8bc7a083a5 P ae8c52b4ed4c4024b60c95d5d9eb2523 [term 1 FOLLOWER]: Refusing update from remote peer 166e3a8113ae46b09a9ae5e3c51e52b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:36:07.518421 25418 consensus_queue.cc:1035] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:36:07.518710 25420 consensus_queue.cc:1035] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:36:07.525923 25424 mvcc.cc:205] Tried to move back new op lower bound from 7333585582144462848 to 7333585581989056512. Current Snapshot: MvccSnapshot[applied={T|T < 7333585582144462848}]
I20260926 12:36:07.527134 25423 mvcc.cc:205] Tried to move back new op lower bound from 7333585582144462848 to 7333585581989056512. Current Snapshot: MvccSnapshot[applied={T|T < 7333585582144462848}]
I20260926 12:36:07.535000 24939 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:36:07.535202 24939 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:36:07.537974 24939 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50: Sending DeleteTablet for 3 replicas of tablet 4f77897b176547f786bc8f8bc7a083a5
I20260926 12:36:07.538800 25202 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f77897b176547f786bc8f8bc7a083a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:36:07 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:60988
I20260926 12:36:07.538900 25064 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f77897b176547f786bc8f8bc7a083a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:36:07 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:59530
I20260926 12:36:07.539093 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 24980
I20260926 12:36:07.539340 25334 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f77897b176547f786bc8f8bc7a083a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:36:07 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:45778
I20260926 12:36:07.540373 25434 tablet_replica.cc:330] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7: stopping tablet replica
I20260926 12:36:07.540652 25434 raft_consensus.cc:2229] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:36:07.540957 25434 raft_consensus.cc:2258] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:36:07.541306 25435 tablet_replica.cc:330] T 4f77897b176547f786bc8f8bc7a083a5 P ae8c52b4ed4c4024b60c95d5d9eb2523: stopping tablet replica
I20260926 12:36:07.541559 25435 raft_consensus.cc:2229] T 4f77897b176547f786bc8f8bc7a083a5 P ae8c52b4ed4c4024b60c95d5d9eb2523 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:36:07.541800 25435 raft_consensus.cc:2258] T 4f77897b176547f786bc8f8bc7a083a5 P ae8c52b4ed4c4024b60c95d5d9eb2523 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:36:07.543097 25434 ts_tablet_manager.cc:1905] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:36:07.545604 25434 ts_tablet_manager.cc:1918] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:36:07.545717 25434 log.cc:1196] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/4f77897b176547f786bc8f8bc7a083a5
I20260926 12:36:07.546017 25434 ts_tablet_manager.cc:1939] T 4f77897b176547f786bc8f8bc7a083a5 P 166e3a8113ae46b09a9ae5e3c51e52b7: Deleting consensus metadata
I20260926 12:36:07.547262 25435 ts_tablet_manager.cc:1905] T 4f77897b176547f786bc8f8bc7a083a5 P ae8c52b4ed4c4024b60c95d5d9eb2523: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:36:07.547647 24929 catalog_manager.cc:4841] TS 166e3a8113ae46b09a9ae5e3c51e52b7 (127.10.98.2:35275): tablet 4f77897b176547f786bc8f8bc7a083a5 (table test-table [id=36345759f03c47979ec09e5813efcc66]) successfully deleted
W20260926 12:36:07.547950 24926 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.98.1:34783 (error 108)
I20260926 12:36:07.548812 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 25135
I20260926 12:36:07.549009 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:07.548716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:34783 (local address 127.0.0.1:59540)
0926 12:36:07.548853 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:07.548925 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.1:34783: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 12:36:07.549463 24926 catalog_manager.cc:4584] TS 1037e58f7ead4eaca6e4ce5ade5fe1c3 (127.10.98.1:34783): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4f77897b176547f786bc8f8bc7a083a5: Network error: Client connection negotiation failed: client connection to 127.10.98.1:34783: connect: Connection refused (error 111)
I20260926 12:36:07.551687 25435 ts_tablet_manager.cc:1918] T 4f77897b176547f786bc8f8bc7a083a5 P ae8c52b4ed4c4024b60c95d5d9eb2523: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:36:07.551816 25435 log.cc:1196] T 4f77897b176547f786bc8f8bc7a083a5 P ae8c52b4ed4c4024b60c95d5d9eb2523: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/4f77897b176547f786bc8f8bc7a083a5
I20260926 12:36:07.552182 25435 ts_tablet_manager.cc:1939] T 4f77897b176547f786bc8f8bc7a083a5 P ae8c52b4ed4c4024b60c95d5d9eb2523: Deleting consensus metadata
I20260926 12:36:07.553738 24926 catalog_manager.cc:4841] TS ae8c52b4ed4c4024b60c95d5d9eb2523 (127.10.98.3:36863): tablet 4f77897b176547f786bc8f8bc7a083a5 (table test-table [id=36345759f03c47979ec09e5813efcc66]) successfully deleted
I20260926 12:36:07.555325 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 25270
I20260926 12:36:07.561986 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 24909
I20260926 12:36:07.569767 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:38593
--webserver_interface=127.10.98.62
--webserver_port=44687
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:43409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:38593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:36:07.683245 25437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:07.683544 25437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:07.683593 25437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:07.687888 25437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:36:07.687978 25437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:07.687996 25437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:07.688011 25437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:36:07.688025 25437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:36:07.693858 25437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:43409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:38593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:38593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=44687
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:07.695024 25437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:07.696075 25437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:07.705134 25442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:07.705034 25443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:07.705240 25437 server_base.cc:1034] running on GCE node
W20260926 12:36:07.705421 25445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:07.706033 25437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:07.707311 25437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:07.708498 25437 hybrid_clock.cc:648] HybridClock initialized: now 1790426167708474 us; error 37 us; skew 500 ppm
Sep 26 12:36:07 dist-test-slave-t7tk krb5kdc[24360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426167, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:07.711585 25437 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:36:07.712589 25437 webserver.cc:460] Webserver started at http://127.10.98.62:44687/ using document root <none> and password file <none>
I20260926 12:36:07.713209 25437 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:07.713286 25437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:07.717701 25437 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:36:07.720124 25452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:07.721736 25437 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:36:07.721894 25437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "50447deb385645bc844b65f2b1b82e50"
format_stamp: "Formatted at 2026-09-26 12:36:05 on dist-test-slave-t7tk"
server_key: "f9ec68586b7e22de42d79a4c826dcf90"
server_key_iv: "e53b8c9686f716fb28c6bae5c6f3630f"
server_key_version: "encryptionkey@0"
I20260926 12:36:07.722601 25437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:07.749166 25437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:07.752326 25437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:07.752521 25437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:07.759447 25437 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:38593
I20260926 12:36:07.759706 25504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:38593 every 8 connection(s)
I20260926 12:36:07.760581 25437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:36:07.764967 25505 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:36:07.766741 25505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50: Bootstrap starting.
I20260926 12:36:07.766918 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 25437
I20260926 12:36:07.767343 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:34783
--local_ip_for_outbound_sockets=127.10.98.1
--tserver_master_addrs=127.10.98.62:38593
--webserver_port=36253
--webserver_interface=127.10.98.1
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:43409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:36:07.784264 25505 log.cc:824] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50: Log is configured to *not* fsync() on all Append() calls
I20260926 12:36:07.796176 25505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:36:07.796592 25505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50: Bootstrap complete.
I20260926 12:36:07.801259 25505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50447deb385645bc844b65f2b1b82e50" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38593 } }
I20260926 12:36:07.801738 25505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50447deb385645bc844b65f2b1b82e50, State: Initialized, Role: FOLLOWER
I20260926 12:36:07.802304 25505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50447deb385645bc844b65f2b1b82e50" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38593 } }
I20260926 12:36:07.802453 25505 raft_consensus.cc:388] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:36:07.802531 25505 raft_consensus.cc:482] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:36:07.802664 25505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 2 FOLLOWER]: Advancing to term 3
I20260926 12:36:07.804795 25505 raft_consensus.cc:504] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50447deb385645bc844b65f2b1b82e50" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38593 } }
I20260926 12:36:07.805195 25505 leader_election.cc:302] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 50447deb385645bc844b65f2b1b82e50; no voters:
I20260926 12:36:07.805573 25505 leader_election.cc:288] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 12:36:07.805689 25509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 12:36:07.805902 25509 raft_consensus.cc:686] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [term 3 LEADER]: Becoming Leader. State: Replica: 50447deb385645bc844b65f2b1b82e50, State: Running, Role: LEADER
I20260926 12:36:07.806221 25509 consensus_queue.cc:237] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50447deb385645bc844b65f2b1b82e50" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38593 } }
I20260926 12:36:07.806918 25505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:36:07.809222 25510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "50447deb385645bc844b65f2b1b82e50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50447deb385645bc844b65f2b1b82e50" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38593 } } }
I20260926 12:36:07.809362 25510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [sys.catalog]: This master's current role is: LEADER
I20260926 12:36:07.809787 25511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 50447deb385645bc844b65f2b1b82e50. Latest consensus state: current_term: 3 leader_uuid: "50447deb385645bc844b65f2b1b82e50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "50447deb385645bc844b65f2b1b82e50" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38593 } } }
I20260926 12:36:07.809921 25511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 50447deb385645bc844b65f2b1b82e50 [sys.catalog]: This master's current role is: LEADER
I20260926 12:36:07.810204 25519 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:36:07.816452 25519 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:36:07.817361 25519 catalog_manager.cc:1259] Loaded cluster ID: 9d429c5929934c8598e6d4f128415c4b
I20260926 12:36:07.817466 25519 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 12:36:07.819092 25519 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3fd74579ad google::LogMessage::Fail() at ??:0
@ 0x7f3fd745a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f3fd74574de google::LogMessage::Flush() at ??:0
@ 0x7f3fd7458a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f3fe1d96fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f3fe1d97d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f3fe1d94579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f3fe1dd8640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3fd96a93d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f3fd96a9c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f3fd96ab3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f3fd969a512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f3fdacde6db start_thread at ??:0
@ 0x7f3fd5f5561f clone at ??:0
W20260926 12:36:07.911262 25507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:07.911540 25507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:07.911584 25507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:07.915729 25507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:07.915793 25507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:07.915853 25507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:36:07.921237 25507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:43409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:34783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=36253
--tserver_master_addrs=127.10.98.62:38593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:07.922410 25507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:07.923799 25507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:07.934489 25546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:07.934499 25549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:07.934497 25547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:07.935070 25507 server_base.cc:1034] running on GCE node
I20260926 12:36:07.935706 25507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:07.936592 25507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:07.937784 25507 hybrid_clock.cc:648] HybridClock initialized: now 1790426167937766 us; error 32 us; skew 500 ppm
Sep 26 12:36:07 dist-test-slave-t7tk krb5kdc[24360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426167, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:07.940863 25507 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:36:07.941828 25507 webserver.cc:460] Webserver started at http://127.10.98.1:36253/ using document root <none> and password file <none>
I20260926 12:36:07.942409 25507 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:07.942472 25507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:07.946674 25507 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:36:07.949391 25556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:07.951164 25507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:36:07.951308 25507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3"
format_stamp: "Formatted at 2026-09-26 12:36:05 on dist-test-slave-t7tk"
server_key: "86689b1600bb9fca1fd09a8fa3575901"
server_key_iv: "c4474883feb4186613bfb253f83b6d51"
server_key_version: "encryptionkey@0"
I20260926 12:36:07.951900 25507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:07.977830 25507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:07.981175 25507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:07.981361 25507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:07.981963 25507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:36:07.983131 25563 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 12:36:07.988809 25507 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 12:36:07.988894 25507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 12:36:07.989009 25507 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 12:36:07.990867 25507 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 12:36:07.990923 25507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260926 12:36:07.991271 25563 tablet_bootstrap.cc:492] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Bootstrap starting.
I20260926 12:36:08.001062 25507 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:34783
I20260926 12:36:08.001332 25670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:34783 every 8 connection(s)
I20260926 12:36:08.002235 25507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:36:08.005028 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 25507
I20260926 12:36:08.005520 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:35275
--local_ip_for_outbound_sockets=127.10.98.2
--tserver_master_addrs=127.10.98.62:38593
--webserver_port=34669
--webserver_interface=127.10.98.2
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:43409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:36:08.015502 25563 log.cc:824] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Log is configured to *not* fsync() on all Append() calls
I20260926 12:36:08.025720 25563 tablet_bootstrap.cc:492] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:36:08.026201 25563 tablet_bootstrap.cc:492] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Bootstrap complete.
I20260926 12:36:08.027657 25563 ts_tablet_manager.cc:1397] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Time spent bootstrapping tablet: real 0.037s user 0.029s sys 0.004s
I20260926 12:36:08.030838 25563 raft_consensus.cc:348] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:08.031417 25563 raft_consensus.cc:729] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1037e58f7ead4eaca6e4ce5ade5fe1c3, State: Initialized, Role: FOLLOWER
I20260926 12:36:08.031957 25563 consensus_queue.cc:260] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:08.033051 25563 ts_tablet_manager.cc:1428] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
W20260926 12:36:08.088932 25673 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:36:08.004288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.1:54237)
0926 12:36:08.005883 (+ 1595us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:08.005955 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:08.007067 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:08.087765 (+ 80698us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":131,"threads_started":1}
W20260926 12:36:08.089644 25671 heartbeater.cc:644] Failed to heartbeat to 127.10.98.62:38593 (0 consecutive failures): Network error: Failed to ping master at 127.10.98.62:38593: Client connection negotiation failed: client connection to 127.10.98.62:38593: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 12:36:08.090433 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:08.090126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.1:57187)
0926 12:36:08.090265 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:08.090352 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 12:36:08.091180 25673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:08.090959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.1:33567)
0926 12:36:08.091070 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:08.091121 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 12:36:08.091399 25671 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:36:08.123220 25674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:08.123490 25674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:08.123536 25674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:08.127604 25674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:08.127679 25674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:08.127745 25674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:36:08.133423 25674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:43409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:35275
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=34669
--tserver_master_addrs=127.10.98.62:38593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:08.134480 25674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:08.135571 25674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:08.145198 25681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:08.145277 25684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:08.145377 25674 server_base.cc:1034] running on GCE node
W20260926 12:36:08.145267 25682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:08.145903 25674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:08.146520 25674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:08.147696 25674 hybrid_clock.cc:648] HybridClock initialized: now 1790426168147672 us; error 36 us; skew 500 ppm
Sep 26 12:36:08 dist-test-slave-t7tk krb5kdc[24360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426168, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:08.150594 25674 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:36:08.151556 25674 webserver.cc:460] Webserver started at http://127.10.98.2:34669/ using document root <none> and password file <none>
I20260926 12:36:08.152118 25674 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:08.152187 25674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:08.156502 25674 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:36:08.158962 25691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:08.160597 25674 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:36:08.160739 25674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "166e3a8113ae46b09a9ae5e3c51e52b7"
format_stamp: "Formatted at 2026-09-26 12:36:05 on dist-test-slave-t7tk"
server_key: "bd1b7c29607ed9e6e3dd8a43904baac6"
server_key_iv: "2bdb2f3474af461d917405398806e23e"
server_key_version: "encryptionkey@0"
I20260926 12:36:08.161391 25674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:08.191372 25674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:08.194926 25674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:08.195123 25674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:08.195731 25674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:36:08.196578 25674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:36:08.196623 25674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:08.196678 25674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:36:08.196708 25674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:08.205698 25674 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:35275
I20260926 12:36:08.205725 25804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:35275 every 8 connection(s)
I20260926 12:36:08.206750 25674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:36:08.210727 25807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:08.208999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.2:45529)
0926 12:36:08.209560 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:08.209979 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":124,"threads_started":1}
W20260926 12:36:08.211175 25805 heartbeater.cc:644] Failed to heartbeat to 127.10.98.62:38593 (0 consecutive failures): Network error: Failed to ping master at 127.10.98.62:38593: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
I20260926 12:36:08.211781 25807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:08.211541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.2:51397)
0926 12:36:08.211679 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:08.211727 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 12:36:08.212498 25807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:08.212288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.2:40923)
0926 12:36:08.212402 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:08.212447 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260926 12:36:08.212713 25805 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 12:36:08.214602 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 25674
I20260926 12:36:08.214978 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:36863
--local_ip_for_outbound_sockets=127.10.98.3
--tserver_master_addrs=127.10.98.62:38593
--webserver_port=32821
--webserver_interface=127.10.98.3
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:43409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:36:08.325371 25808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:08.325639 25808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:08.325700 25808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:08.329784 25808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:08.329849 25808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:08.329933 25808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:36:08.335264 25808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:43409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:36863
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=32821
--tserver_master_addrs=127.10.98.62:38593
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:08.336331 25808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:08.337406 25808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:08.346983 25814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:08.346968 25813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:08.346968 25816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:08.347366 25808 server_base.cc:1034] running on GCE node
I20260926 12:36:08.347836 25808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:08.348518 25808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:08.349694 25808 hybrid_clock.cc:648] HybridClock initialized: now 1790426168349661 us; error 46 us; skew 500 ppm
Sep 26 12:36:08 dist-test-slave-t7tk krb5kdc[24360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426168, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:08.352353 25808 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:36:08.353291 25808 webserver.cc:460] Webserver started at http://127.10.98.3:32821/ using document root <none> and password file <none>
I20260926 12:36:08.353878 25808 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:08.353945 25808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:08.357884 25808 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:36:08.359912 25823 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:08.361339 25808 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:36:08.361446 25808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523"
format_stamp: "Formatted at 2026-09-26 12:36:05 on dist-test-slave-t7tk"
server_key: "06a6a41ce0be753a2c063f47cfdba23e"
server_key_iv: "d72c0f53385185bb7f809071aca2d4ee"
server_key_version: "encryptionkey@0"
I20260926 12:36:08.362030 25808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:08.387157 25808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:08.390398 25808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:08.390587 25808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:08.391201 25808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:36:08.392038 25808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:36:08.392088 25808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:08.392129 25808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:36:08.392158 25808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:08.401129 25808 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:36863
I20260926 12:36:08.401203 25936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:36863 every 8 connection(s)
I20260926 12:36:08.402185 25808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:36:08.406162 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:08.404474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.3:59273)
0926 12:36:08.404987 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:08.405435 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":128,"threads_started":1}
W20260926 12:36:08.406649 25937 heartbeater.cc:644] Failed to heartbeat to 127.10.98.62:38593 (0 consecutive failures): Network error: Failed to ping master at 127.10.98.62:38593: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
I20260926 12:36:08.407171 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:08.406973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.3:44009)
0926 12:36:08.407072 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:08.407117 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 12:36:08.407720 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:08.407538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.3:47221)
0926 12:36:08.407632 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:08.407673 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 12:36:08.407874 25937 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 12:36:08.411998 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 25808
I20260926 12:36:09.092600 25941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:09.092099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.1:48729)
0926 12:36:09.092429 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:09.092494 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":78,"threads_started":1}
I20260926 12:36:09.213814 25942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:09.213350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.2:47085)
0926 12:36:09.213642 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:09.213707 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":71,"threads_started":1}
I20260926 12:36:09.408922 25943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:09.408411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.3:38583)
0926 12:36:09.408712 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:09.408773 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":100,"threads_started":1}
I20260926 12:36:09.697832 25944 raft_consensus.cc:482] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:36:09.698117 25944 raft_consensus.cc:504] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:09.699193 25944 leader_election.cc:288] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae8c52b4ed4c4024b60c95d5d9eb2523 (127.10.98.3:36863), 166e3a8113ae46b09a9ae5e3c51e52b7 (127.10.98.2:35275)
Sep 26 12:36:09 dist-test-slave-t7tk krb5kdc[24360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426167, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.2@KRBTEST.COM
Sep 26 12:36:09 dist-test-slave-t7tk krb5kdc[24360](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426167, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.3@KRBTEST.COM
I20260926 12:36:09.713017 25945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:09.699442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:36863 (local address 127.10.98.1:43941)
0926 12:36:09.699681 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:09.699698 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:09.699814 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:09.701486 (+ 1672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:09.701493 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:09.701944 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:09.702551 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:09.702569 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:09.703798 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:09.703801 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:09.704686 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:09.704695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:09.705456 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:09.705460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:09.705630 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:09.706200 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:09.706226 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:09.708119 (+ 1893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:09.710186 (+ 2067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:09.710189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:09.710191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:09.710576 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:09.711003 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:09.711008 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:09.711010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:09.711188 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:09.711916 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:09.711923 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:09.712228 (+ 305us) client_negotiation.cc:769] Sending connection context
0926 12:36:09.712473 (+ 245us) client_negotiation.cc:240] Negotiation successful
0926 12:36:09.712822 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":88,"spinlock_wait_cycles":34560,"thread_start_us":73,"threads_started":1}
I20260926 12:36:09.713325 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:09.699457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:35275 (local address 127.10.98.1:44131)
0926 12:36:09.699795 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:09.699809 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:09.699908 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:09.701737 (+ 1829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:09.701740 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:09.701954 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:09.702552 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:09.702568 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:09.703795 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:09.703798 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:09.704684 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:09.704693 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:09.705630 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:09.705632 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:09.705691 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:09.706231 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:09.706243 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:09.707807 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:09.710124 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:09.710130 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:09.710144 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:09.710576 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:09.710979 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:09.710982 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:09.710984 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:09.711188 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:09.711916 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:09.711923 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:09.712228 (+ 305us) client_negotiation.cc:769] Sending connection context
0926 12:36:09.712477 (+ 249us) client_negotiation.cc:240] Negotiation successful
0926 12:36:09.713175 (+ 698us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":561,"spinlock_wait_cycles":128128,"thread_start_us":80,"threads_started":1}
I20260926 12:36:09.713354 25947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:09.699648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:44131 (local address 127.10.98.2:35275)
0926 12:36:09.700052 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:09.700073 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:09.700108 (+ 35us) server_negotiation.cc:407] Connection header received
0926 12:36:09.700751 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:09.700756 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:09.701125 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:09.701421 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:09.702716 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:09.703662 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:09.704793 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:09.705340 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:09.705484 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:09.708043 (+ 2559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:09.708050 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:09.708067 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:09.708100 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:09.710004 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:09.710692 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:09.710698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:09.710704 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:09.710847 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:09.711276 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:09.711279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:09.711281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:09.711791 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:09.711912 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:09.712622 (+ 710us) server_negotiation.cc:299] Negotiation successful
0926 12:36:09.712926 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":287,"thread_start_us":69,"threads_started":1}
I20260926 12:36:09.713974 25948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:09.699788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:43941 (local address 127.10.98.3:36863)
0926 12:36:09.700134 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:09.700152 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:09.700182 (+ 30us) server_negotiation.cc:407] Connection header received
0926 12:36:09.700829 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:09.700835 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:09.701181 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:09.701432 (+ 251us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:09.702728 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:09.703668 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:09.704796 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:09.705330 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:09.705462 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:09.708283 (+ 2821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:09.708287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:09.708298 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:09.708323 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:09.710093 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:09.710692 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:09.710698 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:09.710704 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:09.710847 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:09.711276 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:09.711279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:09.711281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:09.711791 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:09.711912 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:09.713539 (+ 1627us) server_negotiation.cc:299] Negotiation successful
0926 12:36:09.713799 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":95,"threads_started":1}
W20260926 12:36:09.715946 25560 leader_election.cc:341] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 166e3a8113ae46b09a9ae5e3c51e52b7 (127.10.98.2:35275): Not found: Tablet not found: 4f77897b176547f786bc8f8bc7a083a5
W20260926 12:36:09.716161 25557 leader_election.cc:341] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae8c52b4ed4c4024b60c95d5d9eb2523 (127.10.98.3:36863): Not found: Tablet not found: 4f77897b176547f786bc8f8bc7a083a5
I20260926 12:36:09.716228 25557 leader_election.cc:302] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1037e58f7ead4eaca6e4ce5ade5fe1c3; no voters: 166e3a8113ae46b09a9ae5e3c51e52b7, ae8c52b4ed4c4024b60c95d5d9eb2523
I20260926 12:36:09.716450 25944 raft_consensus.cc:2726] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:36:10.093871 25945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:10.093548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.1:39605)
0926 12:36:10.093718 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:10.093783 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 12:36:10.214954 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:10.214553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.2:39241)
0926 12:36:10.214800 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:10.214855 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":78,"threads_started":1}
I20260926 12:36:10.410243 25950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:10.409785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.3:60729)
0926 12:36:10.410026 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:10.410089 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":70,"threads_started":1}
I20260926 12:36:11.095327 25951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:11.094821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.1:52111)
0926 12:36:11.095156 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:11.095226 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":94,"threads_started":1}
I20260926 12:36:11.216220 25952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:11.215764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.2:48409)
0926 12:36:11.216060 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:11.216123 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":80,"threads_started":1}
I20260926 12:36:11.247493 25953 raft_consensus.cc:482] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:36:11.247637 25953 raft_consensus.cc:504] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:11.248137 25953 leader_election.cc:288] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae8c52b4ed4c4024b60c95d5d9eb2523 (127.10.98.3:36863), 166e3a8113ae46b09a9ae5e3c51e52b7 (127.10.98.2:35275)
W20260926 12:36:11.248962 25557 leader_election.cc:341] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae8c52b4ed4c4024b60c95d5d9eb2523 (127.10.98.3:36863): Not found: Tablet not found: 4f77897b176547f786bc8f8bc7a083a5
W20260926 12:36:11.249213 25560 leader_election.cc:341] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 166e3a8113ae46b09a9ae5e3c51e52b7 (127.10.98.2:35275): Not found: Tablet not found: 4f77897b176547f786bc8f8bc7a083a5
I20260926 12:36:11.249332 25560 leader_election.cc:302] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1037e58f7ead4eaca6e4ce5ade5fe1c3; no voters: 166e3a8113ae46b09a9ae5e3c51e52b7, ae8c52b4ed4c4024b60c95d5d9eb2523
I20260926 12:36:11.249513 25953 raft_consensus.cc:2726] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:36:11.411487 25954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:11.411040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.3:57245)
0926 12:36:11.411316 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:11.411378 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":95,"threads_started":1}
I20260926 12:36:12.096838 25955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:12.096376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.1:42117)
0926 12:36:12.096668 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:12.096738 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":78,"threads_started":1}
I20260926 12:36:12.217543 25956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:12.217077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.2:59875)
0926 12:36:12.217356 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:12.217415 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":76,"threads_started":1}
I20260926 12:36:12.412774 25957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:12.412282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.3:35427)
0926 12:36:12.412586 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:12.412653 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":96,"threads_started":1}
I20260926 12:36:13.098272 25958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:13.097801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.1:52581)
0926 12:36:13.098124 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:13.098177 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":80,"threads_started":1}
I20260926 12:36:13.218926 25959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:13.218463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.2:41063)
0926 12:36:13.218759 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:13.218826 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":88,"threads_started":1}
I20260926 12:36:13.414160 25960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:13.413664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.3:45459)
0926 12:36:13.413976 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:13.414050 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":82,"threads_started":1}
I20260926 12:36:14.099555 25961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:14.099100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.1:52593)
0926 12:36:14.099397 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:14.099461 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":84,"threads_started":1}
I20260926 12:36:14.220302 25962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:14.219827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.2:34177)
0926 12:36:14.220131 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:14.220200 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":89,"threads_started":1}
I20260926 12:36:14.415338 25963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:14.414946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.3:54361)
0926 12:36:14.415189 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:14.415244 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":71,"threads_started":1}
I20260926 12:36:14.500435 25964 raft_consensus.cc:482] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:36:14.500576 25964 raft_consensus.cc:504] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:14.501027 25964 leader_election.cc:288] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae8c52b4ed4c4024b60c95d5d9eb2523 (127.10.98.3:36863), 166e3a8113ae46b09a9ae5e3c51e52b7 (127.10.98.2:35275)
W20260926 12:36:14.501825 25557 leader_election.cc:341] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae8c52b4ed4c4024b60c95d5d9eb2523 (127.10.98.3:36863): Not found: Tablet not found: 4f77897b176547f786bc8f8bc7a083a5
W20260926 12:36:14.501932 25560 leader_election.cc:341] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 166e3a8113ae46b09a9ae5e3c51e52b7 (127.10.98.2:35275): Not found: Tablet not found: 4f77897b176547f786bc8f8bc7a083a5
I20260926 12:36:14.502038 25560 leader_election.cc:302] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1037e58f7ead4eaca6e4ce5ade5fe1c3; no voters: 166e3a8113ae46b09a9ae5e3c51e52b7, ae8c52b4ed4c4024b60c95d5d9eb2523
I20260926 12:36:14.502242 25964 raft_consensus.cc:2726] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:36:15.100749 25965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:15.100348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.1:50805)
0926 12:36:15.100587 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:15.100651 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":70,"threads_started":1}
I20260926 12:36:15.221571 25966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:15.221075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.2:35891)
0926 12:36:15.221418 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:15.221479 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":79,"threads_started":1}
I20260926 12:36:15.416689 25967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:15.416189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.3:47663)
0926 12:36:15.416521 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:15.416585 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":83,"threads_started":1}
I20260926 12:36:16.101994 25968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:16.101569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.1:58373)
0926 12:36:16.101841 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:16.101896 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":79,"threads_started":1}
I20260926 12:36:16.222725 25969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:16.222323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.2:41533)
0926 12:36:16.222558 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:16.222612 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":76,"threads_started":1}
I20260926 12:36:16.418047 25970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:16.417537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.3:41001)
0926 12:36:16.417825 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:16.417916 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":78,"threads_started":1}
I20260926 12:36:17.103173 25971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:17.102713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.1:39801)
0926 12:36:17.102972 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:17.103031 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":81,"threads_started":1}
I20260926 12:36:17.223897 25972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:17.223487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.2:57849)
0926 12:36:17.223747 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:17.223804 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":74,"threads_started":1}
I20260926 12:36:17.419370 25973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:17.418883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.3:35971)
0926 12:36:17.419209 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:17.419272 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":73,"threads_started":1}
I20260926 12:36:17.945924 25974 raft_consensus.cc:482] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:36:17.946046 25974 raft_consensus.cc:504] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:17.946463 25974 leader_election.cc:288] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae8c52b4ed4c4024b60c95d5d9eb2523 (127.10.98.3:36863), 166e3a8113ae46b09a9ae5e3c51e52b7 (127.10.98.2:35275)
W20260926 12:36:17.947263 25560 leader_election.cc:341] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 166e3a8113ae46b09a9ae5e3c51e52b7 (127.10.98.2:35275): Not found: Tablet not found: 4f77897b176547f786bc8f8bc7a083a5
W20260926 12:36:17.947377 25557 leader_election.cc:341] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae8c52b4ed4c4024b60c95d5d9eb2523 (127.10.98.3:36863): Not found: Tablet not found: 4f77897b176547f786bc8f8bc7a083a5
I20260926 12:36:17.947440 25557 leader_election.cc:302] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1037e58f7ead4eaca6e4ce5ade5fe1c3; no voters: 166e3a8113ae46b09a9ae5e3c51e52b7, ae8c52b4ed4c4024b60c95d5d9eb2523
I20260926 12:36:17.947597 25974 raft_consensus.cc:2726] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:36:18.104586 25975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:18.104100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.1:60141)
0926 12:36:18.104415 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:18.104484 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":94,"threads_started":1}
I20260926 12:36:18.225191 25976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:18.224709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.2:34579)
0926 12:36:18.224988 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:18.225057 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":74,"threads_started":1}
I20260926 12:36:18.420676 25977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:18.420223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.3:56703)
0926 12:36:18.420510 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:18.420575 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":75,"threads_started":1}
I20260926 12:36:19.105966 25978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:19.105537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.1:42509)
0926 12:36:19.105805 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:19.105867 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":75,"threads_started":1}
I20260926 12:36:19.226451 25979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:19.225994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.2:33375)
0926 12:36:19.226289 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:19.226353 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":76,"threads_started":1}
I20260926 12:36:19.421916 25980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:19.421486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.3:47791)
0926 12:36:19.421765 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:19.421824 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":76,"threads_started":1}
I20260926 12:36:20.107260 25981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:20.106843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.1:43759)
0926 12:36:20.107103 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:20.107164 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
I20260926 12:36:20.227743 25982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:20.227315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.2:48025)
0926 12:36:20.227590 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:20.227647 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":87,"threads_started":1}
I20260926 12:36:20.423257 25983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:20.422714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.3:52639)
0926 12:36:20.423064 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:20.423137 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":98,"threads_started":1}
I20260926 12:36:21.108649 25984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:21.108077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.1:33479)
0926 12:36:21.108389 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:21.108474 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":87,"threads_started":1}
I20260926 12:36:21.229081 25985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:21.228597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.2:48445)
0926 12:36:21.228905 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:21.228972 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":86,"threads_started":1}
I20260926 12:36:21.424531 25986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:21.424085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.3:43113)
0926 12:36:21.424372 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:21.424434 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":82,"threads_started":1}
I20260926 12:36:22.109956 25987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:22.109502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.1:52855)
0926 12:36:22.109787 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:22.109852 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":78,"threads_started":1}
I20260926 12:36:22.217473 25988 raft_consensus.cc:482] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:36:22.217609 25988 raft_consensus.cc:504] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae8c52b4ed4c4024b60c95d5d9eb2523" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36863 } } peers { permanent_uuid: "166e3a8113ae46b09a9ae5e3c51e52b7" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 35275 } } peers { permanent_uuid: "1037e58f7ead4eaca6e4ce5ade5fe1c3" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34783 } }
I20260926 12:36:22.217998 25988 leader_election.cc:288] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ae8c52b4ed4c4024b60c95d5d9eb2523 (127.10.98.3:36863), 166e3a8113ae46b09a9ae5e3c51e52b7 (127.10.98.2:35275)
W20260926 12:36:22.218698 25560 leader_election.cc:341] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 166e3a8113ae46b09a9ae5e3c51e52b7 (127.10.98.2:35275): Not found: Tablet not found: 4f77897b176547f786bc8f8bc7a083a5
W20260926 12:36:22.218796 25557 leader_election.cc:341] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ae8c52b4ed4c4024b60c95d5d9eb2523 (127.10.98.3:36863): Not found: Tablet not found: 4f77897b176547f786bc8f8bc7a083a5
I20260926 12:36:22.218894 25557 leader_election.cc:302] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1037e58f7ead4eaca6e4ce5ade5fe1c3; no voters: 166e3a8113ae46b09a9ae5e3c51e52b7, ae8c52b4ed4c4024b60c95d5d9eb2523
I20260926 12:36:22.219039 25988 raft_consensus.cc:2726] T 4f77897b176547f786bc8f8bc7a083a5 P 1037e58f7ead4eaca6e4ce5ade5fe1c3 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:36:22.230314 25989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:22.229874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.2:52865)
0926 12:36:22.230144 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:22.230211 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":89,"threads_started":1}
I20260926 12:36:22.425829 25990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:22.425386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.3:56047)
0926 12:36:22.425673 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:22.425734 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":89,"threads_started":1}
I20260926 12:36:23.111383 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:23.110947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.1:44443)
0926 12:36:23.111231 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:23.111289 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":85,"threads_started":1}
I20260926 12:36:23.231631 25992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:23.231209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:38593 (local address 127.10.98.2:44715)
0926 12:36:23.231467 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:23.231532 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:38593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":70,"threads_started":1}
I20260926 12:36:23.414542 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 25507
I20260926 12:36:23.419756 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 25674
I20260926 12:36:23.424801 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 25808
2026-09-26T12:36:23Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19472 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:36:23 dist-test-slave-t7tk krb5kdc[25995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:36:23 dist-test-slave-t7tk krb5kdc[25995](info): set up 2 sockets
Sep 26 12:36:23 dist-test-slave-t7tk krb5kdc[25995](info): commencing operation
krb5kdc: starting...
W20260926 12:36:24.386533 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:36:24 dist-test-slave-t7tk krb5kdc[25995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:36:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:36:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:36:24.506330 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:45843
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:45843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:36:24.615116 26008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:24.615391 26008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:24.615432 26008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:24.619432 26008 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 12:36:24.619488 26008 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 12:36:24.619501 26008 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 12:36:24.619513 26008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:36:24.619530 26008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:24.619546 26008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:24.619566 26008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:36:24.619589 26008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:36:24.624884 26008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:45843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:45843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:24.625932 26008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:24.626947 26008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:24.635331 26014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:24.635387 26008 server_base.cc:1034] running on GCE node
W20260926 12:36:24.635350 26013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:24.635334 26016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:24.636049 26008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:24.637159 26008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:24.638337 26008 hybrid_clock.cc:648] HybridClock initialized: now 1790426184638316 us; error 32 us; skew 500 ppm
Sep 26 12:36:24 dist-test-slave-t7tk krb5kdc[25995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426184, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:24.641124 26008 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:36:24.642010 26008 webserver.cc:460] Webserver started at http://127.10.98.62:42621/ using document root <none> and password file <none>
I20260926 12:36:24.642536 26008 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:24.642601 26008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:24.642799 26008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:36:24.644795 26008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "2568fc0630b5453e81552ca093ff29aa"
format_stamp: "Formatted at 2026-09-26 12:36:24 on dist-test-slave-t7tk"
server_key: "93b82556ae67b5330088d8ef50200b26"
server_key_iv: "27fc2ff42cc3c18fbb9793c6ff9e6847"
server_key_version: "encryptionkey@0"
I20260926 12:36:24.645349 26008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "2568fc0630b5453e81552ca093ff29aa"
format_stamp: "Formatted at 2026-09-26 12:36:24 on dist-test-slave-t7tk"
server_key: "93b82556ae67b5330088d8ef50200b26"
server_key_iv: "27fc2ff42cc3c18fbb9793c6ff9e6847"
server_key_version: "encryptionkey@0"
I20260926 12:36:24.649045 26008 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:36:24.651665 26023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:24.653052 26008 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:36:24.653297 26008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "2568fc0630b5453e81552ca093ff29aa"
format_stamp: "Formatted at 2026-09-26 12:36:24 on dist-test-slave-t7tk"
server_key: "93b82556ae67b5330088d8ef50200b26"
server_key_iv: "27fc2ff42cc3c18fbb9793c6ff9e6847"
server_key_version: "encryptionkey@0"
I20260926 12:36:24.653409 26008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:24.678841 26008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:24.682345 26008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:24.682530 26008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:24.689713 26008 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:45843
I20260926 12:36:24.689981 26075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:45843 every 8 connection(s)
I20260926 12:36:24.690737 26008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:36:24.692502 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 26008
I20260926 12:36:24.692627 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:36:24.692909 10632 external_mini_cluster.cc:1428] Setting key b9920f7c844d9f192aa2f2c57a0a210c
I20260926 12:36:24.693966 26076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:36:24.700732 26076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa: Bootstrap starting.
Sep 26 12:36:24 dist-test-slave-t7tk krb5kdc[25995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:36:24.703111 26076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa: Neither blocks nor log segments found. Creating new log.
I20260926 12:36:24.704005 26076 log.cc:824] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa: Log is configured to *not* fsync() on all Append() calls
I20260926 12:36:24.706264 26076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa: No bootstrap required, opened a new log
I20260926 12:36:24.708282 26079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:24.694396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51238 (local address 127.10.98.62:45843)
0926 12:36:24.694816 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:24.694835 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:24.694878 (+ 43us) server_negotiation.cc:407] Connection header received
0926 12:36:24.695721 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:24.695743 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:24.696162 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:24.696544 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:24.697276 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:24.698729 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:24.699677 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:24.700429 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:24.700599 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:24.702995 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:24.703001 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:24.703017 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:24.703051 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:24.705058 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:24.705475 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:24.705480 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:24.705485 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:24.705640 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:24.705907 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:24.705909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:24.705911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:24.706402 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:24.706597 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:24.706814 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 12:36:24.707119 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":124,"threads_started":1}
I20260926 12:36:24.709978 26076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2568fc0630b5453e81552ca093ff29aa" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45843 } }
I20260926 12:36:24.710338 26076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:36:24.710446 26076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2568fc0630b5453e81552ca093ff29aa, State: Initialized, Role: FOLLOWER
I20260926 12:36:24.710847 26076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2568fc0630b5453e81552ca093ff29aa" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45843 } }
I20260926 12:36:24.710965 26076 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:36:24.711009 26076 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:36:24.711084 26076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:36:24.711903 26076 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2568fc0630b5453e81552ca093ff29aa" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45843 } }
I20260926 12:36:24.712199 26076 leader_election.cc:302] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2568fc0630b5453e81552ca093ff29aa; no voters:
I20260926 12:36:24.712469 26076 leader_election.cc:288] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:36:24.712620 26081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:36:24.712898 26081 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 1 LEADER]: Becoming Leader. State: Replica: 2568fc0630b5453e81552ca093ff29aa, State: Running, Role: LEADER
I20260926 12:36:24.713302 26081 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2568fc0630b5453e81552ca093ff29aa" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45843 } }
I20260926 12:36:24.713627 26076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:36:24.714856 26082 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2568fc0630b5453e81552ca093ff29aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2568fc0630b5453e81552ca093ff29aa" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45843 } } }
I20260926 12:36:24.714974 26082 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [sys.catalog]: This master's current role is: LEADER
I20260926 12:36:24.714949 26083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2568fc0630b5453e81552ca093ff29aa. Latest consensus state: current_term: 1 leader_uuid: "2568fc0630b5453e81552ca093ff29aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2568fc0630b5453e81552ca093ff29aa" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45843 } } }
I20260926 12:36:24.715049 26083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [sys.catalog]: This master's current role is: LEADER
W20260926 12:36:24.718358 26096 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:36:24.718427 26096 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:36:24.718545 26091 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:36:24.719465 26091 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:36:24.725338 26091 catalog_manager.cc:1347] Generated new cluster ID: cdc7f733b2b94502999c777f1a754d04
I20260926 12:36:24.725404 26091 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:36:24.747377 26091 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:36:24.748694 26091 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:36:24.755774 26091 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa: Generated new TSK 0
I20260926 12:36:24.756735 26091 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
I20260926 12:36:24.796613 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:45843
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:36:24.907763 26102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:24.908028 26102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:24.908087 26102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:24.912050 26102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:24.912106 26102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:24.912185 26102 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:36:24.917472 26102 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:45843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:24.918488 26102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:24.919502 26102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:24.929281 26107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:24.929275 26102 server_base.cc:1034] running on GCE node
W20260926 12:36:24.929293 26110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:24.929255 26108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:24.929878 26102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:24.930464 26102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:24.931627 26102 hybrid_clock.cc:648] HybridClock initialized: now 1790426184931597 us; error 37 us; skew 500 ppm
Sep 26 12:36:24 dist-test-slave-t7tk krb5kdc[25995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426184, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:24.934403 26102 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:36:24.935250 26102 webserver.cc:460] Webserver started at http://127.10.98.1:35661/ using document root <none> and password file <none>
I20260926 12:36:24.935796 26102 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:24.935856 26102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:24.936038 26102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:36:24.938054 26102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "e27f1f65cee44dfcbff5bdeac6c5e884"
format_stamp: "Formatted at 2026-09-26 12:36:24 on dist-test-slave-t7tk"
server_key: "79e5c6b50dbfd5b57657ac92bf3186e2"
server_key_iv: "e5f9b6a7e3152d1ab8052b209c2b0708"
server_key_version: "encryptionkey@0"
I20260926 12:36:24.938581 26102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "e27f1f65cee44dfcbff5bdeac6c5e884"
format_stamp: "Formatted at 2026-09-26 12:36:24 on dist-test-slave-t7tk"
server_key: "79e5c6b50dbfd5b57657ac92bf3186e2"
server_key_iv: "e5f9b6a7e3152d1ab8052b209c2b0708"
server_key_version: "encryptionkey@0"
I20260926 12:36:24.942309 26102 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 12:36:24.944726 26117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:24.946062 26102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:36:24.946218 26102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "e27f1f65cee44dfcbff5bdeac6c5e884"
format_stamp: "Formatted at 2026-09-26 12:36:24 on dist-test-slave-t7tk"
server_key: "79e5c6b50dbfd5b57657ac92bf3186e2"
server_key_iv: "e5f9b6a7e3152d1ab8052b209c2b0708"
server_key_version: "encryptionkey@0"
I20260926 12:36:24.946302 26102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:24.960206 26102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:24.963480 26102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:24.963639 26102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:24.964205 26102 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:36:24.965003 26102 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:36:24.965049 26102 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:24.965124 26102 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:36:24.965147 26102 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:24.973327 26102 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:45521
I20260926 12:36:24.973377 26230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:45521 every 8 connection(s)
I20260926 12:36:24.974221 26102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:36:24.983215 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 26102
Sep 26 12:36:24 dist-test-slave-t7tk krb5kdc[25995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426184, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:36:24.983332 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:36:24.983711 10632 external_mini_cluster.cc:1428] Setting key 53cfec9f2795ff9f5c7d86b8951bacc8
I20260926 12:36:24.987438 26079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:24.976022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:55811 (local address 127.10.98.62:45843)
0926 12:36:24.976166 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:24.976170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:24.977021 (+ 851us) server_negotiation.cc:407] Connection header received
0926 12:36:24.978284 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:24.978287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:24.978333 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:24.978404 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:24.979523 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:24.980050 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:24.981226 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:24.981472 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:24.981558 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:24.984011 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:24.984014 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:24.984016 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:24.984040 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:24.985485 (+ 1445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:24.985977 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:24.985980 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:24.985981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:24.986059 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:24.986363 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:24.986367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:24.986368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:24.986547 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:24.986648 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:24.987155 (+ 507us) server_negotiation.cc:299] Negotiation successful
0926 12:36:24.987281 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:36:24.988194 26233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:24.976410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.1:55811)
0926 12:36:24.976865 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:24.976909 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:24.978094 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:24.978554 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:24.978561 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:24.978929 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:24.979405 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:24.979418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:24.980167 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:24.980169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:24.981040 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:24.981049 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:24.981623 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:24.981625 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:24.981764 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:24.982269 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:24.982284 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:24.983827 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:24.985601 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:24.985607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:24.985622 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:24.985881 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:24.986141 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:24.986144 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:24.986146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:24.986277 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:24.986673 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:24.986681 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:24.986926 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 12:36:24.987093 (+ 167us) client_negotiation.cc:240] Negotiation successful
0926 12:36:24.987349 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":105,"threads_started":1}
I20260926 12:36:24.989466 26231 heartbeater.cc:347] Connected to a master server at 127.10.98.62:45843
I20260926 12:36:24.989701 26231 heartbeater.cc:464] Registering TS with master...
I20260926 12:36:24.990154 26231 heartbeater.cc:511] Master 127.10.98.62:45843 requested a full tablet report, sending...
I20260926 12:36:24.991588 26040 ts_manager.cc:194] Registered new tserver with Master: e27f1f65cee44dfcbff5bdeac6c5e884 (127.10.98.1:45521)
WARNING: no policy specified for kudu/127.10.98.2@KRBTEST.COM; defaulting to no policy
I20260926 12:36:24.992717 26040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:55811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
I20260926 12:36:25.012336 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:45843
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:36:25.121786 26236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:25.122066 26236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:25.122107 26236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:25.126798 26236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:25.126868 26236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:25.126935 26236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:36:25.132280 26236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:45843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:25.133374 26236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:25.134353 26236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:25.144186 26244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:25.144306 26236 server_base.cc:1034] running on GCE node
W20260926 12:36:25.144248 26241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:25.144223 26242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:25.144840 26236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:25.145560 26236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:25.146766 26236 hybrid_clock.cc:648] HybridClock initialized: now 1790426185146728 us; error 51 us; skew 500 ppm
Sep 26 12:36:25 dist-test-slave-t7tk krb5kdc[25995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426185, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:25.149768 26236 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:36:25.150768 26236 webserver.cc:460] Webserver started at http://127.10.98.2:40049/ using document root <none> and password file <none>
I20260926 12:36:25.151347 26236 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:25.151417 26236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:25.151607 26236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:36:25.153738 26236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "95dc2d99ef124dae885d5b29a29d3afe"
format_stamp: "Formatted at 2026-09-26 12:36:25 on dist-test-slave-t7tk"
server_key: "d5855a4b9c006cd9b6c333a63d256ea4"
server_key_iv: "edc1b8a110e454e65a58dbbb589c89d1"
server_key_version: "encryptionkey@0"
I20260926 12:36:25.154284 26236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "95dc2d99ef124dae885d5b29a29d3afe"
format_stamp: "Formatted at 2026-09-26 12:36:25 on dist-test-slave-t7tk"
server_key: "d5855a4b9c006cd9b6c333a63d256ea4"
server_key_iv: "edc1b8a110e454e65a58dbbb589c89d1"
server_key_version: "encryptionkey@0"
I20260926 12:36:25.158197 26236 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:36:25.160650 26251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:25.162120 26236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:36:25.162253 26236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "95dc2d99ef124dae885d5b29a29d3afe"
format_stamp: "Formatted at 2026-09-26 12:36:25 on dist-test-slave-t7tk"
server_key: "d5855a4b9c006cd9b6c333a63d256ea4"
server_key_iv: "edc1b8a110e454e65a58dbbb589c89d1"
server_key_version: "encryptionkey@0"
I20260926 12:36:25.162356 26236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:25.202250 26236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:25.205438 26236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:25.205641 26236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:25.206243 26236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:36:25.207060 26236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:36:25.207119 26236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:25.207175 26236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:36:25.207201 26236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:25.215606 26236 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:37691
I20260926 12:36:25.215648 26364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:37691 every 8 connection(s)
I20260926 12:36:25.216615 26236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:36:25.219213 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 26236
I20260926 12:36:25.219338 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:36:25.219643 10632 external_mini_cluster.cc:1428] Setting key ffaf7061b62a46f39ce9198c170f448e
Sep 26 12:36:25 dist-test-slave-t7tk krb5kdc[25995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426185, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.98.3@KRBTEST.COM; defaulting to no policy
I20260926 12:36:25.230515 26079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:25.218480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:50711 (local address 127.10.98.62:45843)
0926 12:36:25.218593 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:25.218597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:25.219519 (+ 922us) server_negotiation.cc:407] Connection header received
0926 12:36:25.220901 (+ 1382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:25.220905 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:25.220968 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:25.221035 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:25.222277 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.222805 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.223952 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.224176 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.224263 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:25.226928 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:25.226931 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:25.226933 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:25.226958 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:25.228461 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:25.228971 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:25.228974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:25.228975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:25.229050 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:25.229375 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:25.229379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:25.229380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:25.229560 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:25.229645 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:25.230186 (+ 541us) server_negotiation.cc:299] Negotiation successful
0926 12:36:25.230314 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.3@KRBTEST.COM" created.
I20260926 12:36:25.231361 26367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:25.218853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.2:50711)
0926 12:36:25.219363 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:25.219423 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:25.220699 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:25.221255 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:25.221262 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:25.221628 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:25.222131 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:25.222147 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.222925 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.222928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:25.223831 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:25.223841 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.224317 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.224321 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:25.224489 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:25.225036 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:25.225052 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:25.226765 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:25.228572 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:25.228578 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:25.228594 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:25.228873 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:25.229146 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:25.229149 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:25.229151 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:25.229297 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:25.229666 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:25.229673 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:25.229921 (+ 248us) client_negotiation.cc:769] Sending connection context
0926 12:36:25.230100 (+ 179us) client_negotiation.cc:240] Negotiation successful
0926 12:36:25.230439 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":124,"threads_started":1}
I20260926 12:36:25.232610 26365 heartbeater.cc:347] Connected to a master server at 127.10.98.62:45843
I20260926 12:36:25.232875 26365 heartbeater.cc:464] Registering TS with master...
I20260926 12:36:25.233398 26365 heartbeater.cc:511] Master 127.10.98.62:45843 requested a full tablet report, sending...
I20260926 12:36:25.234419 26040 ts_manager.cc:194] Registered new tserver with Master: 95dc2d99ef124dae885d5b29a29d3afe (127.10.98.2:37691)
I20260926 12:36:25.235169 26040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:50711
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
I20260926 12:36:25.249002 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:45843
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:36:25.365306 26370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:25.365567 26370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:25.365618 26370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:25.369742 26370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:25.369808 26370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:25.369877 26370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:36:25.375241 26370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:45843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:25.376314 26370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:25.377362 26370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:25.386960 26375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:25.386996 26376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:25.387326 26378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:25.387347 26370 server_base.cc:1034] running on GCE node
I20260926 12:36:25.387748 26370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:25.388386 26370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:25.389573 26370 hybrid_clock.cc:648] HybridClock initialized: now 1790426185389555 us; error 33 us; skew 500 ppm
Sep 26 12:36:25 dist-test-slave-t7tk krb5kdc[25995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426185, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:25.392274 26370 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:36:25.393236 26370 webserver.cc:460] Webserver started at http://127.10.98.3:37089/ using document root <none> and password file <none>
I20260926 12:36:25.393795 26370 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:25.393854 26370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:25.394050 26370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:36:25.396118 26370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "f0f6a24566714e358b1e1631adef1bb6"
format_stamp: "Formatted at 2026-09-26 12:36:25 on dist-test-slave-t7tk"
server_key: "ddfe42bf7eb7117ce125025f20d77a58"
server_key_iv: "29728b380d7b015822616c431f24bab7"
server_key_version: "encryptionkey@0"
I20260926 12:36:25.396723 26370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "f0f6a24566714e358b1e1631adef1bb6"
format_stamp: "Formatted at 2026-09-26 12:36:25 on dist-test-slave-t7tk"
server_key: "ddfe42bf7eb7117ce125025f20d77a58"
server_key_iv: "29728b380d7b015822616c431f24bab7"
server_key_version: "encryptionkey@0"
I20260926 12:36:25.400480 26370 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260926 12:36:25.402917 26385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:25.404138 26370 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260926 12:36:25.404261 26370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "f0f6a24566714e358b1e1631adef1bb6"
format_stamp: "Formatted at 2026-09-26 12:36:25 on dist-test-slave-t7tk"
server_key: "ddfe42bf7eb7117ce125025f20d77a58"
server_key_iv: "29728b380d7b015822616c431f24bab7"
server_key_version: "encryptionkey@0"
I20260926 12:36:25.404364 26370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:25.421244 26370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:25.424402 26370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:25.424580 26370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:25.425217 26370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:36:25.426040 26370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:36:25.426084 26370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:25.426124 26370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:36:25.426151 26370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:25.434800 26370 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:34785
I20260926 12:36:25.434842 26498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:34785 every 8 connection(s)
I20260926 12:36:25.435775 26370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:36:25.445438 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 26370
Sep 26 12:36:25 dist-test-slave-t7tk krb5kdc[25995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426185, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:36:25.445621 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:36:25.445953 10632 external_mini_cluster.cc:1428] Setting key f7d46895549d3b56cb0f28750afd5072
I20260926 12:36:25.449652 26079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:25.437568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:45403 (local address 127.10.98.62:45843)
0926 12:36:25.437711 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:25.437717 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:25.438647 (+ 930us) server_negotiation.cc:407] Connection header received
0926 12:36:25.439873 (+ 1226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:25.439876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:25.439923 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:25.439995 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:25.441216 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.441744 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.442933 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.443225 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.443333 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:25.446220 (+ 2887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:25.446225 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:25.446227 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:25.446262 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:25.447700 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:25.448182 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:25.448185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:25.448187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:25.448276 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:25.448581 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:25.448585 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:25.448587 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:25.448769 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:25.448851 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:25.449380 (+ 529us) server_negotiation.cc:299] Negotiation successful
0926 12:36:25.449499 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:36:25.450572 26501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:25.438047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.3:45403)
0926 12:36:25.438497 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:25.438541 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:25.439651 (+ 1110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:25.440149 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:25.440156 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:25.440507 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:25.441011 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:25.441024 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.441846 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.441849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:25.442802 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:25.442812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.443362 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.443365 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:25.443506 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:25.444193 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:25.444214 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:25.446012 (+ 1798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:25.447813 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:25.447819 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:25.447837 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:25.448091 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:25.448357 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:25.448360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:25.448362 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:25.448498 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:25.448863 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:25.448870 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:25.449143 (+ 273us) client_negotiation.cc:769] Sending connection context
0926 12:36:25.449329 (+ 186us) client_negotiation.cc:240] Negotiation successful
0926 12:36:25.449658 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":100,"threads_started":1}
I20260926 12:36:25.451704 26499 heartbeater.cc:347] Connected to a master server at 127.10.98.62:45843
I20260926 12:36:25.451920 26499 heartbeater.cc:464] Registering TS with master...
I20260926 12:36:25.452421 26499 heartbeater.cc:511] Master 127.10.98.62:45843 requested a full tablet report, sending...
I20260926 12:36:25.453578 26040 ts_manager.cc:194] Registered new tserver with Master: f0f6a24566714e358b1e1631adef1bb6 (127.10.98.3:34785)
I20260926 12:36:25.454202 26040 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:45403
I20260926 12:36:25.460129 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:36:25.470009 26079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:25.462040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51242 (local address 127.10.98.62:45843)
0926 12:36:25.462163 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:25.462167 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:25.462383 (+ 216us) server_negotiation.cc:407] Connection header received
0926 12:36:25.462482 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:25.462484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:25.462525 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:25.462593 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:25.463340 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.463850 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.464843 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.465067 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.465194 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:25.466676 (+ 1482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:25.466679 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:25.466681 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:25.466704 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:25.468169 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:25.468615 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:25.468618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:25.468620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:25.468788 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:25.469126 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:25.469130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:25.469132 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:25.469445 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:25.469562 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:25.469783 (+ 221us) server_negotiation.cc:299] Negotiation successful
0926 12:36:25.469885 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:36:25.472993 26040 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51242:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:36:25.476052 26040 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:36:25.490581 26512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:25.485282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:34785 (local address 127.0.0.1:58852)
0926 12:36:25.485493 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:25.485506 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:25.485586 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:25.486594 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:25.486597 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:25.486613 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:36:25.486732 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:25.486739 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.487923 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.487926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:25.489189 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:25.489200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.490010 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.490014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:25.490130 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:25.490174 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 12:36:25.490282 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 12:36:25.490408 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":26,"thread_start_us":95,"threads_started":1}
I20260926 12:36:25.490579 26510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:25.484538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:37691 (local address 127.0.0.1:35858)
0926 12:36:25.484913 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:25.484972 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:25.485187 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:25.485997 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:25.486003 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:25.486028 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:36:25.486178 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:25.486186 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.487594 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.487597 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:25.488962 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:25.488970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.490246 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.490249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:25.490328 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:25.490347 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 12:36:25.490403 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 12:36:25.490454 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":29,"thread_start_us":97,"threads_started":1}
I20260926 12:36:25.490749 26515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:25.485997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53114 (local address 127.10.98.1:45521)
0926 12:36:25.486366 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:25.486381 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:25.486401 (+ 20us) server_negotiation.cc:407] Connection header received
0926 12:36:25.486472 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:25.486477 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:25.486655 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:25.486814 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:36:25.487131 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.488021 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.489426 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.490184 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.490281 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:25.490404 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:25.490507 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 12:36:25.490600 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":85,"threads_started":1}
I20260926 12:36:25.490891 26513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:25.485597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:45521 (local address 127.0.0.1:53114)
0926 12:36:25.485800 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:25.485813 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:25.485889 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:25.486951 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:25.486953 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:25.486967 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:36:25.487027 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:25.487031 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.488149 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.488152 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:25.489320 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:25.489327 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.490286 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.490289 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:25.490380 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:25.490402 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:36:25.490476 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 12:36:25.490550 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":18,"thread_start_us":89,"threads_started":1}
I20260926 12:36:25.491212 26511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:25.484691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35858 (local address 127.10.98.2:37691)
0926 12:36:25.485057 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:25.485069 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:25.485117 (+ 48us) server_negotiation.cc:407] Connection header received
0926 12:36:25.485411 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:25.485418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:25.485639 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:25.485799 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:36:25.486306 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.487462 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.489127 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.489931 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.490034 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:25.490209 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:25.490656 (+ 447us) server_negotiation.cc:299] Negotiation successful
0926 12:36:25.491062 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":140,"threads_started":1}
I20260926 12:36:25.491284 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:25.485529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58852 (local address 127.10.98.3:34785)
0926 12:36:25.486162 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:25.486175 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:25.486197 (+ 22us) server_negotiation.cc:407] Connection header received
0926 12:36:25.486268 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:25.486273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:25.486461 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:25.486624 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:36:25.486834 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.487803 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.489322 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.489897 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.489998 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:25.490172 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:25.490649 (+ 477us) server_negotiation.cc:299] Negotiation successful
0926 12:36:25.491154 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":169,"threads_started":1}
I20260926 12:36:25.493935 26433 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba1db38c7c94a2ea86ef7dcdc46547a (DEFAULT_TABLE table=test-table [id=7ed17d61c0374385b3316168546a8c8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:36:25.493933 26165 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba1db38c7c94a2ea86ef7dcdc46547a (DEFAULT_TABLE table=test-table [id=7ed17d61c0374385b3316168546a8c8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:36:25.493933 26299 tablet_service.cc:1467] Processing CreateTablet for tablet 5ba1db38c7c94a2ea86ef7dcdc46547a (DEFAULT_TABLE table=test-table [id=7ed17d61c0374385b3316168546a8c8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:36:25.494912 26433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba1db38c7c94a2ea86ef7dcdc46547a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:36:25.494943 26299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba1db38c7c94a2ea86ef7dcdc46547a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:36:25.494943 26165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ba1db38c7c94a2ea86ef7dcdc46547a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:36:25.501195 26516 tablet_bootstrap.cc:492] T 5ba1db38c7c94a2ea86ef7dcdc46547a P f0f6a24566714e358b1e1631adef1bb6: Bootstrap starting.
I20260926 12:36:25.502413 26517 tablet_bootstrap.cc:492] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884: Bootstrap starting.
I20260926 12:36:25.504238 26517 tablet_bootstrap.cc:654] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884: Neither blocks nor log segments found. Creating new log.
I20260926 12:36:25.504338 26516 tablet_bootstrap.cc:654] T 5ba1db38c7c94a2ea86ef7dcdc46547a P f0f6a24566714e358b1e1631adef1bb6: Neither blocks nor log segments found. Creating new log.
I20260926 12:36:25.504948 26517 log.cc:824] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884: Log is configured to *not* fsync() on all Append() calls
I20260926 12:36:25.504983 26516 log.cc:824] T 5ba1db38c7c94a2ea86ef7dcdc46547a P f0f6a24566714e358b1e1631adef1bb6: Log is configured to *not* fsync() on all Append() calls
I20260926 12:36:25.505445 26518 tablet_bootstrap.cc:492] T 5ba1db38c7c94a2ea86ef7dcdc46547a P 95dc2d99ef124dae885d5b29a29d3afe: Bootstrap starting.
I20260926 12:36:25.507076 26516 tablet_bootstrap.cc:492] T 5ba1db38c7c94a2ea86ef7dcdc46547a P f0f6a24566714e358b1e1631adef1bb6: No bootstrap required, opened a new log
I20260926 12:36:25.507289 26516 ts_tablet_manager.cc:1397] T 5ba1db38c7c94a2ea86ef7dcdc46547a P f0f6a24566714e358b1e1631adef1bb6: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:36:25.508123 26518 tablet_bootstrap.cc:654] T 5ba1db38c7c94a2ea86ef7dcdc46547a P 95dc2d99ef124dae885d5b29a29d3afe: Neither blocks nor log segments found. Creating new log.
I20260926 12:36:25.508410 26517 tablet_bootstrap.cc:492] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884: No bootstrap required, opened a new log
I20260926 12:36:25.508594 26517 ts_tablet_manager.cc:1397] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260926 12:36:25.508976 26518 log.cc:824] T 5ba1db38c7c94a2ea86ef7dcdc46547a P 95dc2d99ef124dae885d5b29a29d3afe: Log is configured to *not* fsync() on all Append() calls
I20260926 12:36:25.510982 26518 tablet_bootstrap.cc:492] T 5ba1db38c7c94a2ea86ef7dcdc46547a P 95dc2d99ef124dae885d5b29a29d3afe: No bootstrap required, opened a new log
I20260926 12:36:25.511162 26518 ts_tablet_manager.cc:1397] T 5ba1db38c7c94a2ea86ef7dcdc46547a P 95dc2d99ef124dae885d5b29a29d3afe: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:36:25.511106 26516 raft_consensus.cc:348] T 5ba1db38c7c94a2ea86ef7dcdc46547a P f0f6a24566714e358b1e1631adef1bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:36:25.511404 26516 raft_consensus.cc:374] T 5ba1db38c7c94a2ea86ef7dcdc46547a P f0f6a24566714e358b1e1631adef1bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:36:25.511468 26516 raft_consensus.cc:729] T 5ba1db38c7c94a2ea86ef7dcdc46547a P f0f6a24566714e358b1e1631adef1bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f6a24566714e358b1e1631adef1bb6, State: Initialized, Role: FOLLOWER
I20260926 12:36:25.511926 26516 consensus_queue.cc:260] T 5ba1db38c7c94a2ea86ef7dcdc46547a P f0f6a24566714e358b1e1631adef1bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:36:25.512233 26517 raft_consensus.cc:348] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:36:25.512482 26517 raft_consensus.cc:374] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:36:25.512537 26517 raft_consensus.cc:729] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e27f1f65cee44dfcbff5bdeac6c5e884, State: Initialized, Role: FOLLOWER
I20260926 12:36:25.512709 26499 heartbeater.cc:503] Master 127.10.98.62:45843 was elected leader, sending a full tablet report...
I20260926 12:36:25.512941 26516 ts_tablet_manager.cc:1428] T 5ba1db38c7c94a2ea86ef7dcdc46547a P f0f6a24566714e358b1e1631adef1bb6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:36:25.512946 26517 consensus_queue.cc:260] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:36:25.513684 26231 heartbeater.cc:503] Master 127.10.98.62:45843 was elected leader, sending a full tablet report...
I20260926 12:36:25.513927 26517 ts_tablet_manager.cc:1428] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:36:25.514853 26518 raft_consensus.cc:348] T 5ba1db38c7c94a2ea86ef7dcdc46547a P 95dc2d99ef124dae885d5b29a29d3afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:36:25.515069 26518 raft_consensus.cc:374] T 5ba1db38c7c94a2ea86ef7dcdc46547a P 95dc2d99ef124dae885d5b29a29d3afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:36:25.515125 26518 raft_consensus.cc:729] T 5ba1db38c7c94a2ea86ef7dcdc46547a P 95dc2d99ef124dae885d5b29a29d3afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95dc2d99ef124dae885d5b29a29d3afe, State: Initialized, Role: FOLLOWER
I20260926 12:36:25.515522 26518 consensus_queue.cc:260] T 5ba1db38c7c94a2ea86ef7dcdc46547a P 95dc2d99ef124dae885d5b29a29d3afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:36:25.516454 26518 ts_tablet_manager.cc:1428] T 5ba1db38c7c94a2ea86ef7dcdc46547a P 95dc2d99ef124dae885d5b29a29d3afe: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:36:25.516872 26365 heartbeater.cc:503] Master 127.10.98.62:45843 was elected leader, sending a full tablet report...
I20260926 12:36:25.605564 26523 raft_consensus.cc:482] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:36:25.605785 26523 raft_consensus.cc:504] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:36:25.607079 26523 leader_election.cc:288] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95dc2d99ef124dae885d5b29a29d3afe (127.10.98.2:37691), f0f6a24566714e358b1e1631adef1bb6 (127.10.98.3:34785)
I20260926 12:36:25.611435 26525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:25.607318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:37691 (local address 127.10.98.1:49889)
0926 12:36:25.607617 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:25.607632 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:25.607806 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:25.608094 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:25.608097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:25.608115 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:36:25.608250 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:25.608256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.609129 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.609132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:25.610193 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:25.610203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.610999 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.611003 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:25.611102 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:25.611130 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 12:36:25.611201 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 12:36:25.611273 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":126,"threads_started":1}
I20260926 12:36:25.611433 26511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:25.607408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:49889 (local address 127.10.98.2:37691)
0926 12:36:25.607539 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:25.607545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:25.607696 (+ 151us) server_negotiation.cc:407] Connection header received
0926 12:36:25.607931 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:25.607934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:25.607985 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:25.608064 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:36:25.608387 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.609007 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.610325 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.610882 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.610960 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:25.611013 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:25.611226 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 12:36:25.611323 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 12:36:25.611974 26526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:25.607731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:34785 (local address 127.10.98.1:51211)
0926 12:36:25.608068 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:25.608099 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:25.608225 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:25.608479 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:25.608482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:25.608499 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:36:25.608619 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:25.608625 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.609744 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.609746 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:25.610636 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:25.610646 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.611455 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.611458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:25.611690 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:25.611714 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:36:25.611788 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 12:36:25.611839 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260926 12:36:25.611984 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:25.607851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:51211 (local address 127.10.98.3:34785)
0926 12:36:25.607967 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:25.607972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:25.608163 (+ 191us) server_negotiation.cc:407] Connection header received
0926 12:36:25.608338 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:25.608341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:25.608392 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:25.608445 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:36:25.608721 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.609624 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.610761 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.611331 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.611423 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:25.611481 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:25.611810 (+ 329us) server_negotiation.cc:299] Negotiation successful
0926 12:36:25.611883 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:36:25.612234 26319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1db38c7c94a2ea86ef7dcdc46547a" candidate_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95dc2d99ef124dae885d5b29a29d3afe" is_pre_election: true
I20260926 12:36:25.612574 26319 raft_consensus.cc:2454] T 5ba1db38c7c94a2ea86ef7dcdc46547a P 95dc2d99ef124dae885d5b29a29d3afe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e27f1f65cee44dfcbff5bdeac6c5e884 in term 0.
I20260926 12:36:25.612967 26452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1db38c7c94a2ea86ef7dcdc46547a" candidate_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0f6a24566714e358b1e1631adef1bb6" is_pre_election: true
I20260926 12:36:25.613163 26121 leader_election.cc:302] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95dc2d99ef124dae885d5b29a29d3afe, e27f1f65cee44dfcbff5bdeac6c5e884; no voters:
I20260926 12:36:25.613300 26452 raft_consensus.cc:2454] T 5ba1db38c7c94a2ea86ef7dcdc46547a P f0f6a24566714e358b1e1631adef1bb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e27f1f65cee44dfcbff5bdeac6c5e884 in term 0.
I20260926 12:36:25.613508 26523 raft_consensus.cc:2781] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:36:25.613720 26523 raft_consensus.cc:482] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:36:25.613777 26523 raft_consensus.cc:3037] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:36:25.614930 26523 raft_consensus.cc:504] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:36:25.615270 26523 leader_election.cc:288] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 1 election: Requested vote from peers 95dc2d99ef124dae885d5b29a29d3afe (127.10.98.2:37691), f0f6a24566714e358b1e1631adef1bb6 (127.10.98.3:34785)
I20260926 12:36:25.615621 26319 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1db38c7c94a2ea86ef7dcdc46547a" candidate_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95dc2d99ef124dae885d5b29a29d3afe"
I20260926 12:36:25.615698 26452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ba1db38c7c94a2ea86ef7dcdc46547a" candidate_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0f6a24566714e358b1e1631adef1bb6"
I20260926 12:36:25.615736 26319 raft_consensus.cc:3037] T 5ba1db38c7c94a2ea86ef7dcdc46547a P 95dc2d99ef124dae885d5b29a29d3afe [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:36:25.615828 26452 raft_consensus.cc:3037] T 5ba1db38c7c94a2ea86ef7dcdc46547a P f0f6a24566714e358b1e1631adef1bb6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:36:25.616775 26452 raft_consensus.cc:2454] T 5ba1db38c7c94a2ea86ef7dcdc46547a P f0f6a24566714e358b1e1631adef1bb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e27f1f65cee44dfcbff5bdeac6c5e884 in term 1.
I20260926 12:36:25.616780 26319 raft_consensus.cc:2454] T 5ba1db38c7c94a2ea86ef7dcdc46547a P 95dc2d99ef124dae885d5b29a29d3afe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e27f1f65cee44dfcbff5bdeac6c5e884 in term 1.
I20260926 12:36:25.617051 26121 leader_election.cc:302] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e27f1f65cee44dfcbff5bdeac6c5e884, f0f6a24566714e358b1e1631adef1bb6; no voters:
I20260926 12:36:25.617234 26523 raft_consensus.cc:2781] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:36:25.617439 26523 raft_consensus.cc:686] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 LEADER]: Becoming Leader. State: Replica: e27f1f65cee44dfcbff5bdeac6c5e884, State: Running, Role: LEADER
I20260926 12:36:25.617774 26523 consensus_queue.cc:237] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:36:25.621050 26039 catalog_manager.cc:5495] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 reported cstate change: term changed from 0 to 1, leader changed from <none> to e27f1f65cee44dfcbff5bdeac6c5e884 (127.10.98.1). New cstate: current_term: 1 leader_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } health_report { overall_health: HEALTHY } } }
I20260926 12:36:25.645627 26515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:25.642102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53116 (local address 127.10.98.1:45521)
0926 12:36:25.642221 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:25.642224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:25.642238 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:36:25.642392 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:25.642394 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:25.642452 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:25.642542 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:36:25.642841 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.643379 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.644283 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:25.644480 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:25.644546 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:25.644830 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:36:25.645280 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:36:25.645371 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:25.645461 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 12:36:25.645514 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:36:25.651688 26452 raft_consensus.cc:1261] T 5ba1db38c7c94a2ea86ef7dcdc46547a P f0f6a24566714e358b1e1631adef1bb6 [term 1 FOLLOWER]: Refusing update from remote peer e27f1f65cee44dfcbff5bdeac6c5e884: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:36:25.651710 26319 raft_consensus.cc:1261] T 5ba1db38c7c94a2ea86ef7dcdc46547a P 95dc2d99ef124dae885d5b29a29d3afe [term 1 FOLLOWER]: Refusing update from remote peer e27f1f65cee44dfcbff5bdeac6c5e884: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:36:25.652380 26527 consensus_queue.cc:1035] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:36:25.652567 26523 consensus_queue.cc:1035] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:36:25.661540 26531 mvcc.cc:205] Tried to move back new op lower bound from 7333585656421912576 to 7333585656292933632. Current Snapshot: MvccSnapshot[applied={T|T < 7333585656421912576}]
I20260926 12:36:25.668510 26040 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:36:25.668715 26040 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51242:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:36:25.671088 26040 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa: Sending DeleteTablet for 3 replicas of tablet 5ba1db38c7c94a2ea86ef7dcdc46547a
I20260926 12:36:25.671880 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 26102
I20260926 12:36:25.671819 26299 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ba1db38c7c94a2ea86ef7dcdc46547a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:36:25 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:35858
I20260926 12:36:25.671964 26433 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ba1db38c7c94a2ea86ef7dcdc46547a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:36:25 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:58852
I20260926 12:36:25.672605 26540 tablet_replica.cc:330] T 5ba1db38c7c94a2ea86ef7dcdc46547a P 95dc2d99ef124dae885d5b29a29d3afe: stopping tablet replica
I20260926 12:36:25.672931 26540 raft_consensus.cc:2229] T 5ba1db38c7c94a2ea86ef7dcdc46547a P 95dc2d99ef124dae885d5b29a29d3afe [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:36:25.673273 26540 raft_consensus.cc:2258] T 5ba1db38c7c94a2ea86ef7dcdc46547a P 95dc2d99ef124dae885d5b29a29d3afe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:36:25.673626 26541 tablet_replica.cc:330] T 5ba1db38c7c94a2ea86ef7dcdc46547a P f0f6a24566714e358b1e1631adef1bb6: stopping tablet replica
I20260926 12:36:25.673928 26541 raft_consensus.cc:2229] T 5ba1db38c7c94a2ea86ef7dcdc46547a P f0f6a24566714e358b1e1631adef1bb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:36:25.674201 26541 raft_consensus.cc:2258] T 5ba1db38c7c94a2ea86ef7dcdc46547a P f0f6a24566714e358b1e1631adef1bb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:36:25.675559 26540 ts_tablet_manager.cc:1905] T 5ba1db38c7c94a2ea86ef7dcdc46547a P 95dc2d99ef124dae885d5b29a29d3afe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:36:25.676524 26541 ts_tablet_manager.cc:1905] T 5ba1db38c7c94a2ea86ef7dcdc46547a P f0f6a24566714e358b1e1631adef1bb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:36:25.678256 26540 ts_tablet_manager.cc:1918] T 5ba1db38c7c94a2ea86ef7dcdc46547a P 95dc2d99ef124dae885d5b29a29d3afe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:36:25.678344 26540 log.cc:1196] T 5ba1db38c7c94a2ea86ef7dcdc46547a P 95dc2d99ef124dae885d5b29a29d3afe: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/5ba1db38c7c94a2ea86ef7dcdc46547a
I20260926 12:36:25.678615 26541 ts_tablet_manager.cc:1918] T 5ba1db38c7c94a2ea86ef7dcdc46547a P f0f6a24566714e358b1e1631adef1bb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:36:25.678627 26540 ts_tablet_manager.cc:1939] T 5ba1db38c7c94a2ea86ef7dcdc46547a P 95dc2d99ef124dae885d5b29a29d3afe: Deleting consensus metadata
I20260926 12:36:25.678701 26541 log.cc:1196] T 5ba1db38c7c94a2ea86ef7dcdc46547a P f0f6a24566714e358b1e1631adef1bb6: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/5ba1db38c7c94a2ea86ef7dcdc46547a
I20260926 12:36:25.678993 26541 ts_tablet_manager.cc:1939] T 5ba1db38c7c94a2ea86ef7dcdc46547a P f0f6a24566714e358b1e1631adef1bb6: Deleting consensus metadata
W20260926 12:36:25.678988 26027 connection.cc:535] client connection to 127.10.98.1:45521 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 12:36:25.679018 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 26236
W20260926 12:36:25.679114 26027 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 12:36:25.680377 26513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:25.680083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:45521 (local address 127.0.0.1:53120)
0926 12:36:25.680220 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:25.680292 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.1:45521: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260926 12:36:25.680550 26027 catalog_manager.cc:4584] TS e27f1f65cee44dfcbff5bdeac6c5e884 (127.10.98.1:45521): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ba1db38c7c94a2ea86ef7dcdc46547a: Network error: Client connection negotiation failed: client connection to 127.10.98.1:45521: connect: Connection refused (error 111)
I20260926 12:36:25.680780 26027 catalog_manager.cc:4841] TS f0f6a24566714e358b1e1631adef1bb6 (127.10.98.3:34785): tablet 5ba1db38c7c94a2ea86ef7dcdc46547a (table test-table [id=7ed17d61c0374385b3316168546a8c8e]) successfully deleted
I20260926 12:36:25.684950 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 26370
I20260926 12:36:25.685401 26513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:25.685130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:37691 (local address 127.0.0.1:35868)
0926 12:36:25.685283 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:25.685325 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.2:37691: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 12:36:25.690021 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 26008
I20260926 12:36:45.696134 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:45843
--webserver_interface=127.10.98.62
--webserver_port=42621
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:45843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:36:45.807474 26543 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:45.807777 26543 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:45.807819 26543 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:45.811911 26543 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 12:36:45.811975 26543 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 12:36:45.811990 26543 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 12:36:45.812001 26543 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:36:45.812018 26543 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:45.812042 26543 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:45.812080 26543 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:36:45.812106 26543 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:36:45.817483 26543 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:45843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:45843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=42621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26543
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:45.818560 26543 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:45.819612 26543 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:45.833894 26549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:45.833992 26550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:45.833894 26552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:45.835486 26543 server_base.cc:1034] running on GCE node
I20260926 12:36:45.836196 26543 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:45.837513 26543 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:45.838691 26543 hybrid_clock.cc:648] HybridClock initialized: now 1790426205838673 us; error 40 us; skew 500 ppm
Sep 26 12:36:45 dist-test-slave-t7tk krb5kdc[25995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426205, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:45.848776 26543 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:36:45.849833 26543 webserver.cc:460] Webserver started at http://127.10.98.62:42621/ using document root <none> and password file <none>
I20260926 12:36:45.850416 26543 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:45.850486 26543 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:45.854809 26543 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:36:45.857036 26559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:45.858500 26543 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:36:45.858619 26543 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "2568fc0630b5453e81552ca093ff29aa"
format_stamp: "Formatted at 2026-09-26 12:36:24 on dist-test-slave-t7tk"
server_key: "93b82556ae67b5330088d8ef50200b26"
server_key_iv: "27fc2ff42cc3c18fbb9793c6ff9e6847"
server_key_version: "encryptionkey@0"
I20260926 12:36:45.859263 26543 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:45.884554 26543 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:45.888000 26543 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:45.888198 26543 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:45.895334 26543 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:45843
I20260926 12:36:45.895612 26611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:45843 every 8 connection(s)
I20260926 12:36:45.896481 26543 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:36:45.900986 26612 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:36:45.902700 26612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa: Bootstrap starting.
I20260926 12:36:45.903277 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 26543
I20260926 12:36:45.903710 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:45521
--local_ip_for_outbound_sockets=127.10.98.1
--tserver_master_addrs=127.10.98.62:45843
--webserver_port=35661
--webserver_interface=127.10.98.1
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:36:45.924661 26612 log.cc:824] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa: Log is configured to *not* fsync() on all Append() calls
I20260926 12:36:45.935843 26612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:36:45.936285 26612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa: Bootstrap complete.
I20260926 12:36:45.941634 26612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2568fc0630b5453e81552ca093ff29aa" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45843 } }
I20260926 12:36:45.942162 26612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2568fc0630b5453e81552ca093ff29aa, State: Initialized, Role: FOLLOWER
I20260926 12:36:45.942741 26612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2568fc0630b5453e81552ca093ff29aa" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45843 } }
I20260926 12:36:45.942901 26612 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:36:45.942971 26612 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:36:45.943063 26612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 1 FOLLOWER]: Advancing to term 2
I20260926 12:36:45.945564 26612 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2568fc0630b5453e81552ca093ff29aa" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45843 } }
I20260926 12:36:45.945958 26612 leader_election.cc:302] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2568fc0630b5453e81552ca093ff29aa; no voters:
I20260926 12:36:45.946310 26612 leader_election.cc:288] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 12:36:45.946445 26616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 2 FOLLOWER]: Leader election won for term 2
I20260926 12:36:45.946705 26616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 2 LEADER]: Becoming Leader. State: Replica: 2568fc0630b5453e81552ca093ff29aa, State: Running, Role: LEADER
I20260926 12:36:45.947021 26616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2568fc0630b5453e81552ca093ff29aa" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45843 } }
I20260926 12:36:45.947558 26612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:36:45.948419 26617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2568fc0630b5453e81552ca093ff29aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2568fc0630b5453e81552ca093ff29aa" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45843 } } }
I20260926 12:36:45.948537 26617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [sys.catalog]: This master's current role is: LEADER
I20260926 12:36:45.948872 26625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:36:45.948843 26618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2568fc0630b5453e81552ca093ff29aa. Latest consensus state: current_term: 2 leader_uuid: "2568fc0630b5453e81552ca093ff29aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2568fc0630b5453e81552ca093ff29aa" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45843 } } }
I20260926 12:36:45.948918 26618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [sys.catalog]: This master's current role is: LEADER
I20260926 12:36:45.953696 26625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:36:45.954598 26625 catalog_manager.cc:1259] Loaded cluster ID: cdc7f733b2b94502999c777f1a754d04
I20260926 12:36:45.954664 26625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:36:45.957582 26625 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:36:45.958366 26625 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa: Loaded TSK: 0
I20260926 12:36:45.975456 26625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa: Generated new TSK 1
I20260926 12:36:45.975765 26625 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 12:36:46.032572 26614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:46.032841 26614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:46.032904 26614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:46.036964 26614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:46.037034 26614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:46.037140 26614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:36:46.042533 26614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:45521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=35661
--tserver_master_addrs=127.10.98.62:45843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26614
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:46.043583 26614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:46.044576 26614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:46.055409 26639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:46.055608 26614 server_base.cc:1034] running on GCE node
W20260926 12:36:46.055413 26640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:46.055428 26642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:46.056229 26614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:46.056880 26614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:46.058043 26614 hybrid_clock.cc:648] HybridClock initialized: now 1790426206058021 us; error 68 us; skew 500 ppm
Sep 26 12:36:46 dist-test-slave-t7tk krb5kdc[25995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426206, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:46.060976 26614 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:36:46.061997 26614 webserver.cc:460] Webserver started at http://127.10.98.1:35661/ using document root <none> and password file <none>
I20260926 12:36:46.062588 26614 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:46.062673 26614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:46.066685 26614 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:36:46.068809 26649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:46.070106 26614 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:36:46.070210 26614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "e27f1f65cee44dfcbff5bdeac6c5e884"
format_stamp: "Formatted at 2026-09-26 12:36:24 on dist-test-slave-t7tk"
server_key: "79e5c6b50dbfd5b57657ac92bf3186e2"
server_key_iv: "e5f9b6a7e3152d1ab8052b209c2b0708"
server_key_version: "encryptionkey@0"
I20260926 12:36:46.070775 26614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:46.089504 26614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:46.093384 26614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:46.093585 26614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:46.094225 26614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:36:46.095530 26656 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 12:36:46.099676 26614 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 12:36:46.099743 26614 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260926 12:36:46.099797 26614 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 12:36:46.101629 26614 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 12:36:46.101692 26614 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260926 12:36:46.101969 26656 tablet_bootstrap.cc:492] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884: Bootstrap starting.
I20260926 12:36:46.112027 26614 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:45521
I20260926 12:36:46.112102 26763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:45521 every 8 connection(s)
I20260926 12:36:46.113323 26614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:36:46.121155 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 26614
I20260926 12:36:46.121670 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:37691
--local_ip_for_outbound_sockets=127.10.98.2
--tserver_master_addrs=127.10.98.62:45843
--webserver_port=40049
--webserver_interface=127.10.98.2
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:36:46.122285 26656 log.cc:824] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884: Log is configured to *not* fsync() on all Append() calls
Sep 26 12:36:46 dist-test-slave-t7tk krb5kdc[25995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426206, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:36:46.130450 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:46.115693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:50117 (local address 127.10.98.62:45843)
0926 12:36:46.116063 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:46.116102 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:46.116430 (+ 328us) server_negotiation.cc:407] Connection header received
0926 12:36:46.117584 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:46.117595 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:46.117904 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:46.118224 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:46.119403 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.120395 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.121945 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.122675 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.122838 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:46.125714 (+ 2876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:46.125719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:46.125732 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:46.125761 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:46.127347 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:46.127851 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:46.127856 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:46.127860 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:46.127977 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:46.128307 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:46.128309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:46.128311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:46.128655 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:46.128770 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:46.129355 (+ 585us) server_negotiation.cc:299] Negotiation successful
0926 12:36:46.129606 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":114,"threads_started":1}
I20260926 12:36:46.130470 26767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:46.115693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.1:50117)
0926 12:36:46.116259 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:46.116307 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:46.117161 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:46.118300 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:46.118308 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:46.118741 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:46.119223 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:46.119238 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.120555 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.120558 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:46.121807 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:46.121819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.122817 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.122821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:46.123042 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:46.123799 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:46.123820 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:46.125511 (+ 1691us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:46.127467 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:46.127473 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:46.127485 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:46.127735 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:46.128065 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:46.128068 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:46.128070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:46.128212 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:46.128763 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:46.128769 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:46.129014 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 12:36:46.129209 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 12:36:46.129552 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":164,"threads_started":1}
I20260926 12:36:46.131274 26656 tablet_bootstrap.cc:492] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:36:46.131711 26656 tablet_bootstrap.cc:492] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884: Bootstrap complete.
I20260926 12:36:46.132920 26764 heartbeater.cc:347] Connected to a master server at 127.10.98.62:45843
I20260926 12:36:46.133031 26656 ts_tablet_manager.cc:1397] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884: Time spent bootstrapping tablet: real 0.031s user 0.023s sys 0.004s
I20260926 12:36:46.133174 26764 heartbeater.cc:464] Registering TS with master...
I20260926 12:36:46.133646 26764 heartbeater.cc:511] Master 127.10.98.62:45843 requested a full tablet report, sending...
I20260926 12:36:46.135886 26576 ts_manager.cc:194] Registered new tserver with Master: e27f1f65cee44dfcbff5bdeac6c5e884 (127.10.98.1:45521)
I20260926 12:36:46.136023 26656 raft_consensus.cc:348] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:36:46.136564 26656 raft_consensus.cc:729] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e27f1f65cee44dfcbff5bdeac6c5e884, State: Initialized, Role: FOLLOWER
I20260926 12:36:46.137063 26656 consensus_queue.cc:260] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:36:46.138128 26656 ts_tablet_manager.cc:1428] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:36:46.139575 26576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:50117
I20260926 12:36:46.142297 26764 heartbeater.cc:503] Master 127.10.98.62:45843 was elected leader, sending a full tablet report...
Sep 26 12:36:46 dist-test-slave-t7tk krb5kdc[25995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426205, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for kudu/127.10.98.1@KRBTEST.COM
I20260926 12:36:46.151407 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:46.138955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:45521 (local address 127.0.0.1:46770)
0926 12:36:46.140245 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:46.140300 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:46.140456 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:46.141493 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:46.141500 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:46.141998 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:46.142134 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:46.142144 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.143585 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.143589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:46.144384 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:46.144394 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.145265 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.145268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:46.145355 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:46.145927 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:46.145940 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:46.147612 (+ 1672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:46.149589 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:46.149599 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:46.149605 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:46.149903 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:46.150205 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:46.150208 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:46.150210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:46.150305 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:46.150720 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:46.150726 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:46.150863 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 12:36:46.151005 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 12:36:46.151224 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":118,"threads_started":1}
I20260926 12:36:46.151429 26772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:46.139067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.10.98.1:45521)
0926 12:36:46.139794 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:46.139806 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:46.140606 (+ 800us) server_negotiation.cc:407] Connection header received
0926 12:36:46.140693 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:46.140699 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:46.141319 (+ 620us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:46.141464 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:46.142267 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.143458 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.144501 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.145030 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.145162 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:46.147786 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:46.147799 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:46.147815 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:46.147844 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:46.149468 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:46.150018 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:46.150022 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:46.150023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:46.150104 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:46.150383 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:46.150385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:46.150386 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:46.150618 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:46.150714 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:46.151110 (+ 396us) server_negotiation.cc:299] Negotiation successful
0926 12:36:46.151295 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":102,"threads_started":1}
I20260926 12:36:46.152536 26698 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ba1db38c7c94a2ea86ef7dcdc46547a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:36:25 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:46770
I20260926 12:36:46.153616 26773 tablet_replica.cc:330] stopping tablet replica
I20260926 12:36:46.153928 26773 raft_consensus.cc:2229] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:36:46.154168 26773 raft_consensus.cc:2258] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:36:46.155217 26773 ts_tablet_manager.cc:1905] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:36:46.159250 26773 ts_tablet_manager.cc:1918] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:36:46.159345 26773 log.cc:1196] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/wal/wals/5ba1db38c7c94a2ea86ef7dcdc46547a
I20260926 12:36:46.159576 26773 ts_tablet_manager.cc:1939] T 5ba1db38c7c94a2ea86ef7dcdc46547a P e27f1f65cee44dfcbff5bdeac6c5e884: Deleting consensus metadata
I20260926 12:36:46.161541 26563 catalog_manager.cc:4841] TS e27f1f65cee44dfcbff5bdeac6c5e884 (127.10.98.1:45521): tablet 5ba1db38c7c94a2ea86ef7dcdc46547a (table test-table [id=7ed17d61c0374385b3316168546a8c8e]) successfully deleted
W20260926 12:36:46.241317 26769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:46.241595 26769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:46.241639 26769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:46.245791 26769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:46.245857 26769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:46.245916 26769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:36:46.251730 26769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:37691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=40049
--tserver_master_addrs=127.10.98.62:45843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26769
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:46.252764 26769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:46.253841 26769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:46.263525 26778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:46.263558 26781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:46.263907 26769 server_base.cc:1034] running on GCE node
W20260926 12:36:46.263978 26779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:46.264391 26769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:46.265058 26769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:46.266335 26769 hybrid_clock.cc:648] HybridClock initialized: now 1790426206266310 us; error 40 us; skew 500 ppm
Sep 26 12:36:46 dist-test-slave-t7tk krb5kdc[25995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426206, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:46.269081 26769 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:36:46.270143 26769 webserver.cc:460] Webserver started at http://127.10.98.2:40049/ using document root <none> and password file <none>
I20260926 12:36:46.270713 26769 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:46.270807 26769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:46.275023 26769 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 12:36:46.277156 26788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:46.278378 26769 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 12:36:46.278486 26769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "95dc2d99ef124dae885d5b29a29d3afe"
format_stamp: "Formatted at 2026-09-26 12:36:25 on dist-test-slave-t7tk"
server_key: "d5855a4b9c006cd9b6c333a63d256ea4"
server_key_iv: "edc1b8a110e454e65a58dbbb589c89d1"
server_key_version: "encryptionkey@0"
I20260926 12:36:46.279042 26769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:46.298067 26769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:46.301645 26769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:46.301824 26769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:46.302423 26769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:36:46.303296 26769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:36:46.303350 26769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:46.303395 26769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:36:46.303423 26769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:46.311789 26769 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:37691
I20260926 12:36:46.311873 26901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:37691 every 8 connection(s)
I20260926 12:36:46.312812 26769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:36:46.319401 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 26769
I20260926 12:36:46.319918 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:34785
--local_ip_for_outbound_sockets=127.10.98.3
--tserver_master_addrs=127.10.98.62:45843
--webserver_port=37089
--webserver_interface=127.10.98.3
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:36:46 dist-test-slave-t7tk krb5kdc[25995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426206, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:36:46.327320 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:46.314741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:55863 (local address 127.10.98.62:45843)
0926 12:36:46.314891 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:46.314895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:46.315836 (+ 941us) server_negotiation.cc:407] Connection header received
0926 12:36:46.317170 (+ 1334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:46.317174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:46.317225 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:46.317300 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:46.318489 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.319070 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.320718 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.320948 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.321032 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:46.323684 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:46.323697 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:46.323700 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:46.323738 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:46.325280 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:46.325762 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:46.325766 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:46.325768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:46.325864 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:46.326180 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:46.326183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:46.326185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:46.326420 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:46.326525 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:46.326999 (+ 474us) server_negotiation.cc:299] Negotiation successful
0926 12:36:46.327145 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:36:46.328156 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:46.315143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.2:55863)
0926 12:36:46.315678 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:46.315721 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:46.316931 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:46.317479 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:46.317486 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:46.317872 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:46.318350 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:46.318364 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.319212 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.319216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:46.320584 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:46.320596 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.321127 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.321132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:46.321362 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:46.321988 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:46.322004 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:46.323481 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:46.325391 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:46.325396 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:46.325410 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:46.325666 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:46.325957 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:46.325959 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:46.325960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:46.326096 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:46.326514 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:46.326519 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:46.326764 (+ 245us) client_negotiation.cc:769] Sending connection context
0926 12:36:46.326921 (+ 157us) client_negotiation.cc:240] Negotiation successful
0926 12:36:46.327183 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":121,"threads_started":1}
I20260926 12:36:46.329365 26902 heartbeater.cc:347] Connected to a master server at 127.10.98.62:45843
I20260926 12:36:46.329594 26902 heartbeater.cc:464] Registering TS with master...
I20260926 12:36:46.330085 26902 heartbeater.cc:511] Master 127.10.98.62:45843 requested a full tablet report, sending...
I20260926 12:36:46.331094 26576 ts_manager.cc:194] Registered new tserver with Master: 95dc2d99ef124dae885d5b29a29d3afe (127.10.98.2:37691)
I20260926 12:36:46.331655 26576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:55863
W20260926 12:36:46.446995 26905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:36:46.447265 26905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:36:46.447325 26905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:36:46.451824 26905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:36:46.451891 26905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:36:46.451970 26905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:36:46.457664 26905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:34785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=37089
--tserver_master_addrs=127.10.98.62:45843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:36:46.458727 26905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:36:46.459887 26905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:36:46.470012 26913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:46.470021 26911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:36:46.470029 26910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:36:46.470887 26905 server_base.cc:1034] running on GCE node
I20260926 12:36:46.471426 26905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:36:46.472266 26905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:36:46.473443 26905 hybrid_clock.cc:648] HybridClock initialized: now 1790426206473409 us; error 44 us; skew 500 ppm
Sep 26 12:36:46 dist-test-slave-t7tk krb5kdc[25995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426206, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:36:46.477025 26905 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:36:46.478293 26905 webserver.cc:460] Webserver started at http://127.10.98.3:37089/ using document root <none> and password file <none>
I20260926 12:36:46.479022 26905 fs_manager.cc:359] Metadata directory not provided
I20260926 12:36:46.479122 26905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:36:46.484673 26905 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260926 12:36:46.487572 26920 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:46.489413 26905 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:36:46.489563 26905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "f0f6a24566714e358b1e1631adef1bb6"
format_stamp: "Formatted at 2026-09-26 12:36:25 on dist-test-slave-t7tk"
server_key: "ddfe42bf7eb7117ce125025f20d77a58"
server_key_iv: "29728b380d7b015822616c431f24bab7"
server_key_version: "encryptionkey@0"
I20260926 12:36:46.490381 26905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:36:46.528066 26905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:36:46.532122 26905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:36:46.532333 26905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:36:46.533123 26905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:36:46.534219 26905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:36:46.534282 26905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:46.534340 26905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:36:46.534372 26905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:36:46.545720 26905 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:34785
I20260926 12:36:46.545744 27033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:34785 every 8 connection(s)
I20260926 12:36:46.547214 26905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:36:46.547528 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 26905
Sep 26 12:36:46 dist-test-slave-t7tk krb5kdc[25995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426206, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:36:46.558444 26766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:46.548335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58576 (local address 127.10.98.62:45843)
0926 12:36:46.548471 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:46.548475 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:46.548974 (+ 499us) server_negotiation.cc:407] Connection header received
0926 12:36:46.549211 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:46.549215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:46.549272 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:46.549353 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:46.550309 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.551050 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.552311 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.552686 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.552804 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:46.554787 (+ 1983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:46.554791 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:46.554794 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:46.554840 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:46.556464 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:46.557016 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:46.557020 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:46.557022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:46.557145 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:46.557479 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:46.557483 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:46.557485 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:46.557712 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:46.557989 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:46.558174 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 12:36:46.558295 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:36:46.562305 27037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:46.549418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:32857 (local address 127.10.98.62:45843)
0926 12:36:46.549664 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:46.549668 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:46.552457 (+ 2789us) server_negotiation.cc:407] Connection header received
0926 12:36:46.552493 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:46.552495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:46.552536 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:46.552606 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:46.554004 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.554552 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.555879 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.556102 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.556189 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:46.558625 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:46.558629 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:46.558631 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:46.558655 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:46.560319 (+ 1664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:46.560814 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:46.560817 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:46.560818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:46.560900 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:46.561235 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:46.561238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:46.561239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:46.561486 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:46.561566 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:46.562076 (+ 510us) server_negotiation.cc:299] Negotiation successful
0926 12:36:46.562184 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":79,"threads_started":1}
I20260926 12:36:46.563170 27038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:46.550126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.3:32857)
0926 12:36:46.550686 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:46.550740 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:46.552208 (+ 1468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:46.552822 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:46.552830 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:46.553248 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:36:46.553861 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:46.553876 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.554674 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.554677 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:46.555758 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:46.555770 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.556243 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.556248 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:46.556426 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:46.556893 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:36:46.556910 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:36:46.558441 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:36:46.560412 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:46.560417 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:46.560430 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:46.560687 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:46.560976 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:36:46.560978 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:36:46.560979 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:36:46.561152 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:36:46.561573 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:36:46.561579 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:36:46.561850 (+ 271us) client_negotiation.cc:769] Sending connection context
0926 12:36:46.562006 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 12:36:46.562274 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":147,"threads_started":1}
I20260926 12:36:46.564495 27034 heartbeater.cc:347] Connected to a master server at 127.10.98.62:45843
I20260926 12:36:46.564757 27034 heartbeater.cc:464] Registering TS with master...
I20260926 12:36:46.565407 27034 heartbeater.cc:511] Master 127.10.98.62:45843 requested a full tablet report, sending...
I20260926 12:36:46.566509 26576 ts_manager.cc:194] Registered new tserver with Master: f0f6a24566714e358b1e1631adef1bb6 (127.10.98.3:34785)
I20260926 12:36:46.567072 26576 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:32857
I20260926 12:36:46.572438 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:36:46.576589 27037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:46.573049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58592 (local address 127.10.98.62:45843)
0926 12:36:46.573251 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:46.573254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:46.573369 (+ 115us) server_negotiation.cc:407] Connection header received
0926 12:36:46.573535 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:46.573537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:46.573637 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:46.573772 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:36:46.574054 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.574609 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.575516 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.575741 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.575825 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:46.576117 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:36:46.576269 (+ 152us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 12:36:46.576484 (+ 215us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58592: token expired
Metrics: {"server-negotiator.queue_time_us":71}
W20260926 12:36:46.576653 27037 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58592: token expired
W20260926 12:36:46.577032 26504 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.10.98.62:45843)
I20260926 12:36:46.584156 27037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:46.577487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58602 (local address 127.10.98.62:45843)
0926 12:36:46.577590 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:46.577594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:46.577608 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:36:46.577728 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:46.577731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:46.577769 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:46.577830 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:36:46.578432 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.578899 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.579660 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.579859 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.579940 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:46.581246 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:36:46.581250 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:36:46.581253 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:36:46.581279 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:36:46.582619 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:46.583029 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:46.583032 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:46.583033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:46.583106 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:36:46.583372 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:36:46.583376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:36:46.583377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:36:46.583550 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:36:46.583663 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:46.583901 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 12:36:46.584009 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 12:36:46.586764 26576 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58602:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:36:46.590050 26576 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:36:46.598441 26698 tablet_service.cc:1467] Processing CreateTablet for tablet 67428e7f9591494bb1abb1867d8a8434 (DEFAULT_TABLE table=test-table [id=c176999a393543b6b5fdffcd59b816fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:36:46.599126 26698 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67428e7f9591494bb1abb1867d8a8434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:36:46.601845 26656 tablet_bootstrap.cc:492] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884: Bootstrap starting.
I20260926 12:36:46.602694 26771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:46.596866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:34785 (local address 127.0.0.1:53998)
0926 12:36:46.596993 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:46.597010 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:46.597125 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:46.598244 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:46.598248 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:46.598268 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:36:46.598385 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:46.598393 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.599812 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.599816 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:46.601070 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:46.601082 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.602377 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.602380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:46.602482 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:46.602500 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 12:36:46.602539 (+ 39us) client_negotiation.cc:240] Negotiation successful
0926 12:36:46.602598 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":1}
I20260926 12:36:46.602695 27044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:46.597262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:37691 (local address 127.0.0.1:39252)
0926 12:36:46.597521 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:46.597537 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:46.597628 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:46.598488 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:46.598491 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:46.598511 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:36:46.598601 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:46.598669 (+ 68us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.600155 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.600159 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:46.601264 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:46.601272 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.602246 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.602249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:46.602350 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:46.602401 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 12:36:46.602469 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 12:36:46.602553 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"mutex_wait_us":1,"thread_start_us":88,"threads_started":1}
I20260926 12:36:46.602789 27045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:46.597681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39252 (local address 127.10.98.2:37691)
0926 12:36:46.598019 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:46.598032 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:46.598054 (+ 22us) server_negotiation.cc:407] Connection header received
0926 12:36:46.598129 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:46.598134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:46.598346 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:46.598509 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:36:46.598777 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.600024 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.601380 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.602121 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.602249 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:46.602415 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:46.602519 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 12:36:46.602624 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":151,"threads_started":1}
I20260926 12:36:46.602849 26656 tablet_bootstrap.cc:654] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884: Neither blocks nor log segments found. Creating new log.
I20260926 12:36:46.603418 27043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:46.596999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53998 (local address 127.10.98.3:34785)
0926 12:36:46.597810 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:46.597821 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:46.597842 (+ 21us) server_negotiation.cc:407] Connection header received
0926 12:36:46.597911 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:46.597916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:46.598095 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:46.598255 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:36:46.598528 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.599673 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.601249 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.602253 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.602991 (+ 738us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:46.603138 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:46.603187 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 12:36:46.603283 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":702,"thread_start_us":157,"threads_started":1}
I20260926 12:36:46.604286 26656 tablet_bootstrap.cc:492] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884: No bootstrap required, opened a new log
I20260926 12:36:46.604375 26656 ts_tablet_manager.cc:1397] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 12:36:46.604803 26656 raft_consensus.cc:348] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:36:46.604915 26656 raft_consensus.cc:374] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:36:46.604948 26656 raft_consensus.cc:729] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e27f1f65cee44dfcbff5bdeac6c5e884, State: Initialized, Role: FOLLOWER
I20260926 12:36:46.605073 26656 consensus_queue.cc:260] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:36:46.605643 26656 ts_tablet_manager.cc:1428] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 12:36:46.606153 26836 tablet_service.cc:1467] Processing CreateTablet for tablet 67428e7f9591494bb1abb1867d8a8434 (DEFAULT_TABLE table=test-table [id=c176999a393543b6b5fdffcd59b816fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:36:46.606225 26968 tablet_service.cc:1467] Processing CreateTablet for tablet 67428e7f9591494bb1abb1867d8a8434 (DEFAULT_TABLE table=test-table [id=c176999a393543b6b5fdffcd59b816fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:36:46.607076 26836 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67428e7f9591494bb1abb1867d8a8434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:36:46.607091 26968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67428e7f9591494bb1abb1867d8a8434. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:36:46.613961 27047 tablet_bootstrap.cc:492] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe: Bootstrap starting.
W20260926 12:36:46.614627 26765 tablet.cc:2369] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:36:46.615096 27048 tablet_bootstrap.cc:492] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6: Bootstrap starting.
I20260926 12:36:46.616423 27047 tablet_bootstrap.cc:654] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe: Neither blocks nor log segments found. Creating new log.
I20260926 12:36:46.616940 27048 tablet_bootstrap.cc:654] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6: Neither blocks nor log segments found. Creating new log.
I20260926 12:36:46.617188 27047 log.cc:824] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe: Log is configured to *not* fsync() on all Append() calls
I20260926 12:36:46.617974 27048 log.cc:824] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6: Log is configured to *not* fsync() on all Append() calls
I20260926 12:36:46.619149 27047 tablet_bootstrap.cc:492] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe: No bootstrap required, opened a new log
I20260926 12:36:46.619366 27047 ts_tablet_manager.cc:1397] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:36:46.619946 27048 tablet_bootstrap.cc:492] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6: No bootstrap required, opened a new log
I20260926 12:36:46.620132 27048 ts_tablet_manager.cc:1397] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:36:46.623292 27047 raft_consensus.cc:348] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:36:46.623584 27047 raft_consensus.cc:374] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:36:46.623643 27047 raft_consensus.cc:729] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95dc2d99ef124dae885d5b29a29d3afe, State: Initialized, Role: FOLLOWER
I20260926 12:36:46.623849 27048 raft_consensus.cc:348] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:36:46.624104 27048 raft_consensus.cc:374] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:36:46.624158 27048 raft_consensus.cc:729] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f6a24566714e358b1e1631adef1bb6, State: Initialized, Role: FOLLOWER
I20260926 12:36:46.624109 27047 consensus_queue.cc:260] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:36:46.624588 27048 consensus_queue.cc:260] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:36:46.625005 26902 heartbeater.cc:503] Master 127.10.98.62:45843 was elected leader, sending a full tablet report...
I20260926 12:36:46.625196 27047 ts_tablet_manager.cc:1428] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 12:36:46.625564 27048 ts_tablet_manager.cc:1428] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:36:46.625574 27034 heartbeater.cc:503] Master 127.10.98.62:45843 was elected leader, sending a full tablet report...
I20260926 12:36:46.765561 27051 raft_consensus.cc:482] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:36:46.765786 27051 raft_consensus.cc:504] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:36:46.766954 27051 leader_election.cc:288] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0f6a24566714e358b1e1631adef1bb6 (127.10.98.3:34785), e27f1f65cee44dfcbff5bdeac6c5e884 (127.10.98.1:45521)
I20260926 12:36:46.772276 27043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:46.767324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:52743 (local address 127.10.98.3:34785)
0926 12:36:46.767467 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:46.767472 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:46.767911 (+ 439us) server_negotiation.cc:407] Connection header received
0926 12:36:46.768050 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:46.768053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:46.768102 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:46.768196 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:36:46.768526 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.769314 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.771033 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.771649 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.771761 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:46.771828 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:46.772074 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 12:36:46.772145 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 12:36:46.772279 27054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:46.767589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:50951 (local address 127.10.98.1:45521)
0926 12:36:46.767812 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:46.767815 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:46.767889 (+ 74us) server_negotiation.cc:407] Connection header received
0926 12:36:46.768050 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:46.768053 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:46.768119 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:46.768221 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:36:46.768522 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.769328 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.771033 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.771689 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.771776 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:46.771957 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:46.772073 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 12:36:46.772146 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":67,"threads_started":1}
I20260926 12:36:46.772336 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:46.767461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:45521 (local address 127.10.98.2:50951)
0926 12:36:46.767823 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:46.767839 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:46.767962 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:46.768227 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:46.768230 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:46.768253 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:36:46.768426 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:46.768433 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.769433 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.769437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:46.770911 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:46.770922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.771781 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.771784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:46.771897 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:46.771975 (+ 78us) client_negotiation.cc:769] Sending connection context
0926 12:36:46.772056 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 12:36:46.772163 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":171,"spinlock_wait_cycles":24832,"thread_start_us":112,"threads_started":1}
I20260926 12:36:46.772343 27053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:46.767189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:34785 (local address 127.10.98.2:52743)
0926 12:36:46.767823 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:46.767839 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:46.767962 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:46.768226 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:36:46.768229 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:36:46.768253 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:36:46.768428 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:46.768433 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.769433 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.769437 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:46.770911 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:36:46.770922 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.771781 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.771784 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:36:46.771912 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:46.771980 (+ 68us) client_negotiation.cc:769] Sending connection context
0926 12:36:46.772051 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 12:36:46.772163 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"mutex_wait_us":166,"spinlock_wait_cycles":27264}
I20260926 12:36:46.773002 26988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67428e7f9591494bb1abb1867d8a8434" candidate_uuid: "95dc2d99ef124dae885d5b29a29d3afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0f6a24566714e358b1e1631adef1bb6" is_pre_election: true
I20260926 12:36:46.773037 26718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67428e7f9591494bb1abb1867d8a8434" candidate_uuid: "95dc2d99ef124dae885d5b29a29d3afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" is_pre_election: true
I20260926 12:36:46.773329 26988 raft_consensus.cc:2454] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95dc2d99ef124dae885d5b29a29d3afe in term 0.
I20260926 12:36:46.773360 26718 raft_consensus.cc:2454] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95dc2d99ef124dae885d5b29a29d3afe in term 0.
I20260926 12:36:46.773795 26792 leader_election.cc:302] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95dc2d99ef124dae885d5b29a29d3afe, f0f6a24566714e358b1e1631adef1bb6; no voters:
I20260926 12:36:46.774082 27051 raft_consensus.cc:2781] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:36:46.774169 27051 raft_consensus.cc:482] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:36:46.774273 27051 raft_consensus.cc:3037] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:36:46.775352 27051 raft_consensus.cc:504] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:36:46.775663 27051 leader_election.cc:288] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe [CANDIDATE]: Term 1 election: Requested vote from peers f0f6a24566714e358b1e1631adef1bb6 (127.10.98.3:34785), e27f1f65cee44dfcbff5bdeac6c5e884 (127.10.98.1:45521)
I20260926 12:36:46.775995 26988 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67428e7f9591494bb1abb1867d8a8434" candidate_uuid: "95dc2d99ef124dae885d5b29a29d3afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0f6a24566714e358b1e1631adef1bb6"
I20260926 12:36:46.776039 26718 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67428e7f9591494bb1abb1867d8a8434" candidate_uuid: "95dc2d99ef124dae885d5b29a29d3afe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884"
I20260926 12:36:46.776104 26988 raft_consensus.cc:3037] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:36:46.776154 26718 raft_consensus.cc:3037] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:36:46.777069 26718 raft_consensus.cc:2454] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95dc2d99ef124dae885d5b29a29d3afe in term 1.
I20260926 12:36:46.777220 26988 raft_consensus.cc:2454] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95dc2d99ef124dae885d5b29a29d3afe in term 1.
I20260926 12:36:46.777431 26792 leader_election.cc:302] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95dc2d99ef124dae885d5b29a29d3afe, e27f1f65cee44dfcbff5bdeac6c5e884; no voters:
I20260926 12:36:46.777585 27051 raft_consensus.cc:2781] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:36:46.777761 27051 raft_consensus.cc:686] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe [term 1 LEADER]: Becoming Leader. State: Replica: 95dc2d99ef124dae885d5b29a29d3afe, State: Running, Role: LEADER
I20260926 12:36:46.778065 27051 consensus_queue.cc:237] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:36:46.780920 26575 catalog_manager.cc:5495] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe reported cstate change: term changed from 0 to 1, leader changed from <none> to 95dc2d99ef124dae885d5b29a29d3afe (127.10.98.2). New cstate: current_term: 1 leader_uuid: "95dc2d99ef124dae885d5b29a29d3afe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } health_report { overall_health: UNKNOWN } } }
I20260926 12:36:46.797998 27045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:36:46.794542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39260 (local address 127.10.98.2:37691)
0926 12:36:46.794690 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 12:36:46.794694 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:36:46.794715 (+ 21us) server_negotiation.cc:407] Connection header received
0926 12:36:46.794789 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:36:46.794792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:36:46.794844 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:36:46.794945 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:36:46.795185 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.795747 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.796598 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:36:46.796817 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:36:46.796897 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:36:46.797184 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:36:46.797626 (+ 442us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:36:46.797721 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 12:36:46.797809 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 12:36:46.797862 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 12:36:46.799497 27035 tablet.cc:2369] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:36:46.803752 26718 raft_consensus.cc:1261] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Refusing update from remote peer 95dc2d99ef124dae885d5b29a29d3afe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:36:46.803757 26988 raft_consensus.cc:1261] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6 [term 1 FOLLOWER]: Refusing update from remote peer 95dc2d99ef124dae885d5b29a29d3afe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:36:46.804384 27051 consensus_queue.cc:1035] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:36:46.804579 27055 consensus_queue.cc:1035] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:36:46.810366 27059 mvcc.cc:205] Tried to move back new op lower bound from 7333585743061200896 to 7333585742965612544. Current Snapshot: MvccSnapshot[applied={T|T < 7333585743061200896}]
I20260926 12:36:46.814468 27060 mvcc.cc:205] Tried to move back new op lower bound from 7333585743061200896 to 7333585742965612544. Current Snapshot: MvccSnapshot[applied={T|T < 7333585743061200896}]
I20260926 12:36:46.818281 26576 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:36:46.818424 26576 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58602:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:36:46.820590 26576 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa: Sending DeleteTablet for 3 replicas of tablet 67428e7f9591494bb1abb1867d8a8434
I20260926 12:36:46.821273 26968 tablet_service.cc:1514] Processing DeleteTablet for tablet 67428e7f9591494bb1abb1867d8a8434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:36:46 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:53998
I20260926 12:36:46.821337 26698 tablet_service.cc:1514] Processing DeleteTablet for tablet 67428e7f9591494bb1abb1867d8a8434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:36:46 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:46770
I20260926 12:36:46.821398 26836 tablet_service.cc:1514] Processing DeleteTablet for tablet 67428e7f9591494bb1abb1867d8a8434 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:36:46 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:39252
I20260926 12:36:46.821666 27067 tablet_replica.cc:330] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884: stopping tablet replica
I20260926 12:36:46.821762 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 26614
I20260926 12:36:46.821785 27067 raft_consensus.cc:2229] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:36:46.823130 27068 tablet_replica.cc:330] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6: stopping tablet replica
I20260926 12:36:46.823454 27068 raft_consensus.cc:2229] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:36:46.823755 27068 raft_consensus.cc:2258] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:36:46.824118 27069 tablet_replica.cc:330] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe: stopping tablet replica
I20260926 12:36:46.824409 27069 raft_consensus.cc:2229] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:36:46.824728 27069 raft_consensus.cc:2258] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:36:46.825147 27068 ts_tablet_manager.cc:1905] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:36:46.826138 27069 ts_tablet_manager.cc:1905] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:36:46.827412 27068 ts_tablet_manager.cc:1918] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:36:46.827497 27068 log.cc:1196] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/67428e7f9591494bb1abb1867d8a8434
I20260926 12:36:46.827770 27068 ts_tablet_manager.cc:1939] T 67428e7f9591494bb1abb1867d8a8434 P f0f6a24566714e358b1e1631adef1bb6: Deleting consensus metadata
I20260926 12:36:46.828719 27069 ts_tablet_manager.cc:1918] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:36:46.828811 27069 log.cc:1196] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/67428e7f9591494bb1abb1867d8a8434
W20260926 12:36:46.828920 26563 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.98.1:45521 (error 108)
I20260926 12:36:46.829136 27069 ts_tablet_manager.cc:1939] T 67428e7f9591494bb1abb1867d8a8434 P 95dc2d99ef124dae885d5b29a29d3afe: Deleting consensus metadata
I20260926 12:36:46.829468 26563 catalog_manager.cc:4841] TS f0f6a24566714e358b1e1631adef1bb6 (127.10.98.3:34785): tablet 67428e7f9591494bb1abb1867d8a8434 (table test-table [id=c176999a393543b6b5fdffcd59b816fe]) successfully deleted
I20260926 12:36:46.830202 26563 catalog_manager.cc:4841] TS 95dc2d99ef124dae885d5b29a29d3afe (127.10.98.2:37691): tablet 67428e7f9591494bb1abb1867d8a8434 (table test-table [id=c176999a393543b6b5fdffcd59b816fe]) successfully deleted
W20260926 12:36:46.830942 26771 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:36:46.829776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:45521 (local address 127.0.0.1:46778)
0926 12:36:46.829885 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 12:36:46.829899 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:36:46.829996 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:36:46.830853 (+ 857us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.1:45521: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 12:36:46.831015 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 26769
W20260926 12:36:46.831137 26563 catalog_manager.cc:4584] TS e27f1f65cee44dfcbff5bdeac6c5e884 (127.10.98.1:45521): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 67428e7f9591494bb1abb1867d8a8434: Network error: Client connection negotiation failed: client connection to 127.10.98.1:45521: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 12:36:46.836340 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 26905
I20260926 12:36:46.841071 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 26543
I20260926 12:37:06.847680 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:45843
--webserver_interface=127.10.98.62
--webserver_port=42621
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:45843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:06.959764 27071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:06.960052 27071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:06.960095 27071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:06.964188 27071 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 12:37:06.964250 27071 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 12:37:06.964265 27071 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 12:37:06.964278 27071 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:06.964295 27071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:06.964311 27071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:06.964326 27071 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:06.964345 27071 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:06.969775 27071 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:45843
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:45843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=42621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27071
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:06.970827 27071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:06.971786 27071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:06.980005 27076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:06.980010 27077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:06.980046 27079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:06.980249 27071 server_base.cc:1034] running on GCE node
I20260926 12:37:06.980844 27071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:06.981985 27071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:06.983153 27071 hybrid_clock.cc:648] HybridClock initialized: now 1790426226983116 us; error 51 us; skew 500 ppm
Sep 26 12:37:06 dist-test-slave-t7tk krb5kdc[25995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426226, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:06.986054 27071 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:37:06.986995 27071 webserver.cc:460] Webserver started at http://127.10.98.62:42621/ using document root <none> and password file <none>
I20260926 12:37:06.987525 27071 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:06.987597 27071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:06.991528 27071 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:37:06.993796 27086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:06.995234 27071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:37:06.995357 27071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "2568fc0630b5453e81552ca093ff29aa"
format_stamp: "Formatted at 2026-09-26 12:36:24 on dist-test-slave-t7tk"
server_key: "93b82556ae67b5330088d8ef50200b26"
server_key_iv: "27fc2ff42cc3c18fbb9793c6ff9e6847"
server_key_version: "encryptionkey@0"
I20260926 12:37:06.995923 27071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:07.006263 27071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:07.009879 27071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:07.010061 27071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:07.016678 27071 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:45843
I20260926 12:37:07.016958 27138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:45843 every 8 connection(s)
I20260926 12:37:07.017769 27071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:07.022117 27139 sys_catalog.cc:263] Verifying existing consensus state
I20260926 12:37:07.023833 27139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa: Bootstrap starting.
I20260926 12:37:07.024812 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 27071
I20260926 12:37:07.025273 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:45521
--local_ip_for_outbound_sockets=127.10.98.1
--tserver_master_addrs=127.10.98.62:45843
--webserver_port=35661
--webserver_interface=127.10.98.1
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:37:07.041321 27139 log.cc:824] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:07.052624 27139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:37:07.053011 27139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa: Bootstrap complete.
I20260926 12:37:07.056849 27139 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2568fc0630b5453e81552ca093ff29aa" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45843 } }
I20260926 12:37:07.057299 27139 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2568fc0630b5453e81552ca093ff29aa, State: Initialized, Role: FOLLOWER
I20260926 12:37:07.057809 27139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2568fc0630b5453e81552ca093ff29aa" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45843 } }
I20260926 12:37:07.057935 27139 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:07.058004 27139 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:07.058099 27139 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 2 FOLLOWER]: Advancing to term 3
I20260926 12:37:07.060072 27139 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2568fc0630b5453e81552ca093ff29aa" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45843 } }
I20260926 12:37:07.060382 27139 leader_election.cc:302] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2568fc0630b5453e81552ca093ff29aa; no voters:
I20260926 12:37:07.060714 27139 leader_election.cc:288] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 12:37:07.060863 27143 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 3 FOLLOWER]: Leader election won for term 3
I20260926 12:37:07.061139 27143 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [term 3 LEADER]: Becoming Leader. State: Replica: 2568fc0630b5453e81552ca093ff29aa, State: Running, Role: LEADER
I20260926 12:37:07.061460 27143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2568fc0630b5453e81552ca093ff29aa" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45843 } }
I20260926 12:37:07.062065 27139 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:07.062914 27144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2568fc0630b5453e81552ca093ff29aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2568fc0630b5453e81552ca093ff29aa" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45843 } } }
I20260926 12:37:07.062956 27145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2568fc0630b5453e81552ca093ff29aa. Latest consensus state: current_term: 3 leader_uuid: "2568fc0630b5453e81552ca093ff29aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2568fc0630b5453e81552ca093ff29aa" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45843 } } }
I20260926 12:37:07.063058 27144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:07.063078 27145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:07.063414 27152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:07.068497 27152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:07.069340 27152 catalog_manager.cc:1259] Loaded cluster ID: cdc7f733b2b94502999c777f1a754d04
I20260926 12:37:07.069404 27152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:07.071874 27152 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:07.072623 27152 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa: Loaded TSK: 0
I20260926 12:37:07.072705 27152 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2568fc0630b5453e81552ca093ff29aa: Loaded TSK: 1
F20260926 12:37:07.073609 27152 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4150d699ad google::LogMessage::Fail() at ??:0
@ 0x7f4150d6c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f4150d694de google::LogMessage::Flush() at ??:0
@ 0x7f4150d6aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4153a2d18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f4153a25f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f415b6ce7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f415b6a899c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f415b6a9178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f415b6eadf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f415b70982b std::function<>::operator()() at ??:0
@ 0x7f415b6a8b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f415b6a9f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f415b6a6579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f415b6ea640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4152fbb3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4152fbbc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4152fbd3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4152fac512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f41545f06db start_thread at ??:0
@ 0x7f414f86761f clone at ??:0
W20260926 12:37:07.152403 27141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:07.152695 27141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:07.152740 27141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:07.156973 27141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:07.157022 27141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:07.157083 27141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:37:07.162693 27141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:45521
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=35661
--tserver_master_addrs=127.10.98.62:45843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:07.163751 27141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:07.164940 27141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:07.174839 27186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:07.174839 27187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:07.175005 27141 server_base.cc:1034] running on GCE node
W20260926 12:37:07.174877 27189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:07.175599 27141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:07.176262 27141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:07.177440 27141 hybrid_clock.cc:648] HybridClock initialized: now 1790426227177414 us; error 42 us; skew 500 ppm
Sep 26 12:37:07 dist-test-slave-t7tk krb5kdc[25995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426227, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:07.180168 27141 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:37:07.181169 27141 webserver.cc:460] Webserver started at http://127.10.98.1:35661/ using document root <none> and password file <none>
I20260926 12:37:07.181710 27141 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:07.181777 27141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:07.185637 27141 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:37:07.187724 27196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:07.188978 27141 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:37:07.189075 27141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "e27f1f65cee44dfcbff5bdeac6c5e884"
format_stamp: "Formatted at 2026-09-26 12:36:24 on dist-test-slave-t7tk"
server_key: "79e5c6b50dbfd5b57657ac92bf3186e2"
server_key_iv: "e5f9b6a7e3152d1ab8052b209c2b0708"
server_key_version: "encryptionkey@0"
I20260926 12:37:07.189648 27141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:07.224051 27141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:07.227494 27141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:07.227674 27141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:07.228320 27141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:37:07.229578 27203 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 12:37:07.235157 27141 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 12:37:07.235236 27141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260926 12:37:07.235308 27141 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 12:37:07.237087 27141 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 12:37:07.237169 27141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260926 12:37:07.237438 27203 tablet_bootstrap.cc:492] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884: Bootstrap starting.
I20260926 12:37:07.247350 27141 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:45521
I20260926 12:37:07.247406 27310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:45521 every 8 connection(s)
I20260926 12:37:07.248641 27141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:37:07.251943 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 27141
I20260926 12:37:07.252461 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:37691
--local_ip_for_outbound_sockets=127.10.98.2
--tserver_master_addrs=127.10.98.62:45843
--webserver_port=40049
--webserver_interface=127.10.98.2
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:37:07.258376 27203 log.cc:824] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:07.268083 27203 tablet_bootstrap.cc:492] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 12:37:07.268524 27203 tablet_bootstrap.cc:492] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884: Bootstrap complete.
I20260926 12:37:07.269889 27203 ts_tablet_manager.cc:1397] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884: Time spent bootstrapping tablet: real 0.033s user 0.019s sys 0.012s
I20260926 12:37:07.273092 27203 raft_consensus.cc:348] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:37:07.273665 27203 raft_consensus.cc:729] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e27f1f65cee44dfcbff5bdeac6c5e884, State: Initialized, Role: FOLLOWER
I20260926 12:37:07.274296 27203 consensus_queue.cc:260] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:37:07.275377 27203 ts_tablet_manager.cc:1428] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
W20260926 12:37:07.296867 27313 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:37:07.250806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.1:54571)
0926 12:37:07.251343 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:07.251428 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:07.252542 (+ 1114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:07.296115 (+ 43573us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":126,"threads_started":1}
W20260926 12:37:07.297464 27311 heartbeater.cc:644] Failed to heartbeat to 127.10.98.62:45843 (0 consecutive failures): Network error: Failed to ping master at 127.10.98.62:45843: Client connection negotiation failed: client connection to 127.10.98.62:45843: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 12:37:07.298207 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:07.297957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.1:40937)
0926 12:37:07.298082 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:07.298148 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 12:37:07.298919 27313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:07.298677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.1:45167)
0926 12:37:07.298819 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:07.298867 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 12:37:07.299104 27311 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:37:07.373311 27314 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:07.373564 27314 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:07.373610 27314 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:07.377796 27314 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:07.377856 27314 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:07.377916 27314 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:37:07.383260 27314 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:37691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=40049
--tserver_master_addrs=127.10.98.62:45843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27314
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:07.384303 27314 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:07.385342 27314 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:07.395179 27321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:07.395220 27322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:07.395328 27324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:07.395233 27314 server_base.cc:1034] running on GCE node
I20260926 12:37:07.395888 27314 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:07.396502 27314 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:07.397665 27314 hybrid_clock.cc:648] HybridClock initialized: now 1790426227397634 us; error 43 us; skew 500 ppm
Sep 26 12:37:07 dist-test-slave-t7tk krb5kdc[25995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426227, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:07.400444 27314 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:37:07.401362 27314 webserver.cc:460] Webserver started at http://127.10.98.2:40049/ using document root <none> and password file <none>
I20260926 12:37:07.401955 27314 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:07.402025 27314 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:07.405951 27314 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:37:07.408113 27331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:07.409477 27314 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:37:07.409585 27314 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "95dc2d99ef124dae885d5b29a29d3afe"
format_stamp: "Formatted at 2026-09-26 12:36:25 on dist-test-slave-t7tk"
server_key: "d5855a4b9c006cd9b6c333a63d256ea4"
server_key_iv: "edc1b8a110e454e65a58dbbb589c89d1"
server_key_version: "encryptionkey@0"
I20260926 12:37:07.410132 27314 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:07.435537 27314 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:07.438824 27314 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:07.438990 27314 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:07.439607 27314 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:37:07.440439 27314 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:37:07.440488 27314 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:07.440531 27314 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:37:07.440558 27314 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:07.449316 27314 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:37691
I20260926 12:37:07.449347 27444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:37691 every 8 connection(s)
I20260926 12:37:07.450315 27314 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:37:07.453264 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 27314
I20260926 12:37:07.453747 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:34785
--local_ip_for_outbound_sockets=127.10.98.3
--tserver_master_addrs=127.10.98.62:45843
--webserver_port=37089
--webserver_interface=127.10.98.3
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:37:07.454806 27447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:07.452529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.2:32963)
0926 12:37:07.453117 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:07.453267 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":116,"threads_started":1}
W20260926 12:37:07.455278 27445 heartbeater.cc:644] Failed to heartbeat to 127.10.98.62:45843 (0 consecutive failures): Network error: Failed to ping master at 127.10.98.62:45843: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
I20260926 12:37:07.455927 27447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:07.455664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.2:56869)
0926 12:37:07.455795 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:07.455855 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 12:37:07.456660 27447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:07.456450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.2:45077)
0926 12:37:07.456555 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:07.456604 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 12:37:07.456816 27445 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 12:37:07.567464 27448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:07.567747 27448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:07.567807 27448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:07.571925 27448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:07.571990 27448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:07.572074 27448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:37:07.577564 27448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:34785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=37089
--tserver_master_addrs=127.10.98.62:45843
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27448
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:07.578649 27448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:07.579653 27448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:07.589679 27456 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:07.589700 27454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:07.589679 27453 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:07.589875 27448 server_base.cc:1034] running on GCE node
I20260926 12:37:07.590371 27448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:07.591112 27448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:07.592362 27448 hybrid_clock.cc:648] HybridClock initialized: now 1790426227592331 us; error 44 us; skew 500 ppm
Sep 26 12:37:07 dist-test-slave-t7tk krb5kdc[25995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426227, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:07.601014 27448 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:37:07.602084 27448 webserver.cc:460] Webserver started at http://127.10.98.3:37089/ using document root <none> and password file <none>
I20260926 12:37:07.602728 27448 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:07.602797 27448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:07.607038 27448 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260926 12:37:07.609236 27463 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:07.610541 27448 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 12:37:07.610673 27448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "f0f6a24566714e358b1e1631adef1bb6"
format_stamp: "Formatted at 2026-09-26 12:36:25 on dist-test-slave-t7tk"
server_key: "ddfe42bf7eb7117ce125025f20d77a58"
server_key_iv: "29728b380d7b015822616c431f24bab7"
server_key_version: "encryptionkey@0"
I20260926 12:37:07.611274 27448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:07.622172 27448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:07.625177 27448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:07.625363 27448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:07.625932 27448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:37:07.626739 27448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:37:07.626792 27448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:07.626844 27448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:37:07.626875 27448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:07.635552 27448 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:34785
I20260926 12:37:07.635599 27576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:34785 every 8 connection(s)
I20260926 12:37:07.636534 27448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:37:07.640419 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:07.638773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.3:47837)
0926 12:37:07.639221 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:07.639705 (+ 484us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":106,"threads_started":1}
W20260926 12:37:07.640914 27577 heartbeater.cc:644] Failed to heartbeat to 127.10.98.62:45843 (0 consecutive failures): Network error: Failed to ping master at 127.10.98.62:45843: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
I20260926 12:37:07.640986 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 27448
I20260926 12:37:07.641912 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:07.641298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.3:54353)
0926 12:37:07.641803 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:07.641857 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":442}
I20260926 12:37:07.642565 27579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:07.642308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.3:37471)
0926 12:37:07.642469 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:07.642515 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260926 12:37:07.642725 27577 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 12:37:08.300289 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:08.299756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.1:36985)
0926 12:37:08.300112 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:08.300186 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":85,"threads_started":1}
I20260926 12:37:08.431813 27582 raft_consensus.cc:482] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:37:08.432024 27582 raft_consensus.cc:504] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:37:08.433089 27582 leader_election.cc:288] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0f6a24566714e358b1e1631adef1bb6 (127.10.98.3:34785), 95dc2d99ef124dae885d5b29a29d3afe (127.10.98.2:37691)
Sep 26 12:37:08 dist-test-slave-t7tk krb5kdc[25995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426227, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.3@KRBTEST.COM
Sep 26 12:37:08 dist-test-slave-t7tk krb5kdc[25995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426227, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.2@KRBTEST.COM
I20260926 12:37:08.446883 27584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:08.433612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:37691 (local address 127.10.98.1:54671)
0926 12:37:08.433862 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:08.433874 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:08.433964 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:08.436184 (+ 2220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:08.436187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:08.436358 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:08.436630 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:08.436646 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:08.437940 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:08.437942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:08.438809 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:08.438818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:08.439496 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:08.439500 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:08.439739 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:08.440426 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:37:08.440443 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:08.442385 (+ 1942us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:08.444813 (+ 2428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:08.444814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:08.444816 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:08.444959 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:08.445317 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:08.445320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:08.445322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:08.445407 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:08.445966 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:08.445968 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:08.446166 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 12:37:08.446329 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 12:37:08.446694 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":169,"spinlock_wait_cycles":8960,"thread_start_us":64,"threads_started":1}
I20260926 12:37:08.446897 27583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:08.433499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:44871 (local address 127.10.98.3:34785)
0926 12:37:08.433940 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:08.433956 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:08.434003 (+ 47us) server_negotiation.cc:407] Connection header received
0926 12:37:08.434902 (+ 899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:08.434908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:08.435379 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:08.435711 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:08.436792 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:08.437797 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:08.438921 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:08.439357 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:08.439508 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:08.442323 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:08.442328 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:08.442341 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:08.442368 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:08.444306 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:08.444817 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:08.444822 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:08.444826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:08.444938 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:08.445341 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:08.445345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:08.445346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:08.445780 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:08.445880 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:08.446460 (+ 580us) server_negotiation.cc:299] Negotiation successful
0926 12:37:08.446715 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":102,"threads_started":1}
I20260926 12:37:08.446911 27585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:08.433783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:54671 (local address 127.10.98.2:37691)
0926 12:37:08.434120 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:08.434130 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:08.434160 (+ 30us) server_negotiation.cc:407] Connection header received
0926 12:37:08.434898 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:08.434907 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:08.435379 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:08.435711 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:08.436798 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:08.437798 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:08.438919 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:08.439367 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:08.439537 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:08.442579 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:08.442586 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:08.442600 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:08.442634 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:08.444306 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:08.445054 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:08.445059 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:08.445064 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:08.445214 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:08.445498 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:08.445500 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:08.445501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:08.445855 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:08.445980 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:08.446468 (+ 488us) server_negotiation.cc:299] Negotiation successful
0926 12:37:08.446731 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":133,"threads_started":1}
I20260926 12:37:08.446913 27581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:08.433333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:34785 (local address 127.10.98.1:44871)
0926 12:37:08.433509 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:08.433522 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:08.433632 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:08.435770 (+ 2138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:08.435774 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:08.436127 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:08.436629 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:08.436645 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:08.437952 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:08.437955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:08.438807 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:08.438817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:08.439478 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:08.439481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:08.439663 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:08.440459 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:37:08.440471 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:08.442094 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:08.444462 (+ 2368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:08.444468 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:08.444479 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:08.444723 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:08.445072 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:08.445074 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:08.445075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:08.445237 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:08.445894 (+ 657us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:08.445900 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:08.446168 (+ 268us) client_negotiation.cc:769] Sending connection context
0926 12:37:08.446373 (+ 205us) client_negotiation.cc:240] Negotiation successful
0926 12:37:08.446766 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":291,"spinlock_wait_cycles":2176}
W20260926 12:37:08.449431 27200 leader_election.cc:341] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 95dc2d99ef124dae885d5b29a29d3afe (127.10.98.2:37691): Not found: Tablet not found: 67428e7f9591494bb1abb1867d8a8434
W20260926 12:37:08.449666 27200 leader_election.cc:341] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0f6a24566714e358b1e1631adef1bb6 (127.10.98.3:34785): Not found: Tablet not found: 67428e7f9591494bb1abb1867d8a8434
I20260926 12:37:08.449730 27200 leader_election.cc:302] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e27f1f65cee44dfcbff5bdeac6c5e884; no voters: 95dc2d99ef124dae885d5b29a29d3afe, f0f6a24566714e358b1e1631adef1bb6
I20260926 12:37:08.449975 27582 raft_consensus.cc:2726] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:37:08.457692 27586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:08.457305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.2:56103)
0926 12:37:08.457546 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:08.457605 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":64,"threads_started":1}
I20260926 12:37:08.643752 27587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:08.643312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.3:60887)
0926 12:37:08.643587 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:08.643649 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":85,"threads_started":1}
I20260926 12:37:09.301378 27588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:09.300951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.1:45379)
0926 12:37:09.301229 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:09.301283 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":73,"threads_started":1}
I20260926 12:37:09.458953 27589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:09.458442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.2:54803)
0926 12:37:09.458747 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:09.458807 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":74,"threads_started":1}
I20260926 12:37:09.644867 27590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:09.644444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.3:43541)
0926 12:37:09.644704 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:09.644760 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":85,"threads_started":1}
I20260926 12:37:10.302609 27591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:10.302197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.1:57761)
0926 12:37:10.302450 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:10.302515 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":76,"threads_started":1}
I20260926 12:37:10.386423 27592 raft_consensus.cc:482] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:37:10.386579 27592 raft_consensus.cc:504] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:37:10.386955 27592 leader_election.cc:288] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0f6a24566714e358b1e1631adef1bb6 (127.10.98.3:34785), 95dc2d99ef124dae885d5b29a29d3afe (127.10.98.2:37691)
W20260926 12:37:10.387815 27200 leader_election.cc:341] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0f6a24566714e358b1e1631adef1bb6 (127.10.98.3:34785): Not found: Tablet not found: 67428e7f9591494bb1abb1867d8a8434
W20260926 12:37:10.387969 27200 leader_election.cc:341] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 95dc2d99ef124dae885d5b29a29d3afe (127.10.98.2:37691): Not found: Tablet not found: 67428e7f9591494bb1abb1867d8a8434
I20260926 12:37:10.388006 27200 leader_election.cc:302] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e27f1f65cee44dfcbff5bdeac6c5e884; no voters: 95dc2d99ef124dae885d5b29a29d3afe, f0f6a24566714e358b1e1631adef1bb6
I20260926 12:37:10.388159 27592 raft_consensus.cc:2726] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:37:10.460275 27593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:10.459752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.2:59935)
0926 12:37:10.460090 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:10.460156 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":90,"threads_started":1}
I20260926 12:37:10.646051 27594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:10.645644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.3:37515)
0926 12:37:10.645882 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:10.645953 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
I20260926 12:37:11.303751 27595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:11.303308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.1:40077)
0926 12:37:11.303593 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:11.303652 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":82,"threads_started":1}
I20260926 12:37:11.461491 27596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:11.461045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.2:41723)
0926 12:37:11.461325 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:11.461385 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":107,"threads_started":1}
I20260926 12:37:11.647285 27597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:11.646829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.3:50559)
0926 12:37:11.647127 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:11.647188 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":82,"threads_started":1}
I20260926 12:37:11.973212 27598 raft_consensus.cc:482] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:37:11.973451 27598 raft_consensus.cc:504] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:37:11.973834 27598 leader_election.cc:288] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0f6a24566714e358b1e1631adef1bb6 (127.10.98.3:34785), 95dc2d99ef124dae885d5b29a29d3afe (127.10.98.2:37691)
W20260926 12:37:11.974593 27200 leader_election.cc:341] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0f6a24566714e358b1e1631adef1bb6 (127.10.98.3:34785): Not found: Tablet not found: 67428e7f9591494bb1abb1867d8a8434
W20260926 12:37:11.974754 27200 leader_election.cc:341] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 95dc2d99ef124dae885d5b29a29d3afe (127.10.98.2:37691): Not found: Tablet not found: 67428e7f9591494bb1abb1867d8a8434
I20260926 12:37:11.974825 27200 leader_election.cc:302] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e27f1f65cee44dfcbff5bdeac6c5e884; no voters: 95dc2d99ef124dae885d5b29a29d3afe, f0f6a24566714e358b1e1631adef1bb6
I20260926 12:37:11.974969 27598 raft_consensus.cc:2726] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:37:12.305305 27599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:12.304743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.1:55513)
0926 12:37:12.305140 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:12.305207 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":122,"threads_started":1}
I20260926 12:37:12.462697 27600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:12.462266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.2:56271)
0926 12:37:12.462542 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:12.462600 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":93,"threads_started":1}
I20260926 12:37:12.648562 27601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:12.648109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.3:41681)
0926 12:37:12.648358 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:12.648467 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":73,"threads_started":1}
I20260926 12:37:13.306591 27602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:13.306133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.1:45925)
0926 12:37:13.306420 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:13.306492 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":73,"threads_started":1}
I20260926 12:37:13.464047 27603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:13.463579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.2:53501)
0926 12:37:13.463881 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:13.463946 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":90,"threads_started":1}
I20260926 12:37:13.649842 27604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:13.649351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.3:54347)
0926 12:37:13.649677 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:13.649743 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":100,"threads_started":1}
I20260926 12:37:14.307796 27605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:14.307327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.1:45221)
0926 12:37:14.307630 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:14.307691 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":99,"threads_started":1}
I20260926 12:37:14.465440 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:14.464826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.2:54597)
0926 12:37:14.465224 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:14.465286 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":89,"threads_started":1}
I20260926 12:37:14.651103 27607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:14.650609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.3:59207)
0926 12:37:14.650935 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:14.651009 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":84,"threads_started":1}
I20260926 12:37:15.309248 27608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:15.308686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.1:51245)
0926 12:37:15.309035 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:15.309129 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":92,"threads_started":1}
I20260926 12:37:15.370647 27609 raft_consensus.cc:482] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:37:15.370798 27609 raft_consensus.cc:504] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:37:15.371225 27609 leader_election.cc:288] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0f6a24566714e358b1e1631adef1bb6 (127.10.98.3:34785), 95dc2d99ef124dae885d5b29a29d3afe (127.10.98.2:37691)
W20260926 12:37:15.372051 27200 leader_election.cc:341] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0f6a24566714e358b1e1631adef1bb6 (127.10.98.3:34785): Not found: Tablet not found: 67428e7f9591494bb1abb1867d8a8434
W20260926 12:37:15.372184 27200 leader_election.cc:341] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 95dc2d99ef124dae885d5b29a29d3afe (127.10.98.2:37691): Not found: Tablet not found: 67428e7f9591494bb1abb1867d8a8434
I20260926 12:37:15.372231 27200 leader_election.cc:302] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e27f1f65cee44dfcbff5bdeac6c5e884; no voters: 95dc2d99ef124dae885d5b29a29d3afe, f0f6a24566714e358b1e1631adef1bb6
I20260926 12:37:15.372411 27609 raft_consensus.cc:2726] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:37:15.466681 27610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:15.466239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.2:57821)
0926 12:37:15.466519 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:15.466582 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":74,"threads_started":1}
I20260926 12:37:15.652269 27611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:15.651825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.3:42257)
0926 12:37:15.652111 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:15.652171 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":88,"threads_started":1}
I20260926 12:37:16.310503 27612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:16.310071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.1:39787)
0926 12:37:16.310325 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:16.310396 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
I20260926 12:37:16.468092 27613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:16.467613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.2:56619)
0926 12:37:16.467927 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:16.467991 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":86,"threads_started":1}
I20260926 12:37:16.653540 27614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:16.653073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.3:49297)
0926 12:37:16.653382 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:16.653442 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":75,"threads_started":1}
I20260926 12:37:17.311884 27615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:17.311432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.1:37941)
0926 12:37:17.311718 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:17.311780 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":82,"threads_started":1}
I20260926 12:37:17.469460 27616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:17.469022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.2:36987)
0926 12:37:17.469294 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:17.469362 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":94,"threads_started":1}
I20260926 12:37:17.654816 27617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:17.654400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.3:46311)
0926 12:37:17.654653 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:17.654715 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":70,"threads_started":1}
I20260926 12:37:18.313199 27618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:18.312678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.1:55283)
0926 12:37:18.312995 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:18.313056 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":91,"threads_started":1}
I20260926 12:37:18.470712 27619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:18.470277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.2:33397)
0926 12:37:18.470557 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:18.470617 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":74,"threads_started":1}
I20260926 12:37:18.655989 27620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:18.655593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.3:47083)
0926 12:37:18.655833 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:18.655890 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":70,"threads_started":1}
I20260926 12:37:19.314421 27621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:19.314022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.1:46841)
0926 12:37:19.314265 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:19.314326 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260926 12:37:19.471845 27622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:19.471423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.2:45567)
0926 12:37:19.471694 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:19.471750 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":77,"threads_started":1}
I20260926 12:37:19.657305 27623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:19.656778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.3:33311)
0926 12:37:19.657135 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:19.657203 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":130,"threads_started":1}
I20260926 12:37:20.315769 27624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:20.315264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.1:56023)
0926 12:37:20.315571 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:20.315649 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":77,"threads_started":1}
I20260926 12:37:20.473224 27625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:20.472674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.2:35869)
0926 12:37:20.473009 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:20.473086 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":77,"threads_started":1}
I20260926 12:37:20.658725 27626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:20.658189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.3:44483)
0926 12:37:20.658550 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:20.658610 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":99,"threads_started":1}
I20260926 12:37:20.912467 27627 raft_consensus.cc:482] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:37:20.912597 27627 raft_consensus.cc:504] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0f6a24566714e358b1e1631adef1bb6" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 34785 } } peers { permanent_uuid: "95dc2d99ef124dae885d5b29a29d3afe" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37691 } } peers { permanent_uuid: "e27f1f65cee44dfcbff5bdeac6c5e884" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 45521 } }
I20260926 12:37:20.912967 27627 leader_election.cc:288] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f0f6a24566714e358b1e1631adef1bb6 (127.10.98.3:34785), 95dc2d99ef124dae885d5b29a29d3afe (127.10.98.2:37691)
W20260926 12:37:20.913810 27200 leader_election.cc:341] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f0f6a24566714e358b1e1631adef1bb6 (127.10.98.3:34785): Not found: Tablet not found: 67428e7f9591494bb1abb1867d8a8434
W20260926 12:37:20.913925 27200 leader_election.cc:341] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 95dc2d99ef124dae885d5b29a29d3afe (127.10.98.2:37691): Not found: Tablet not found: 67428e7f9591494bb1abb1867d8a8434
I20260926 12:37:20.913965 27200 leader_election.cc:302] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e27f1f65cee44dfcbff5bdeac6c5e884; no voters: 95dc2d99ef124dae885d5b29a29d3afe, f0f6a24566714e358b1e1631adef1bb6
I20260926 12:37:20.914100 27627 raft_consensus.cc:2726] T 67428e7f9591494bb1abb1867d8a8434 P e27f1f65cee44dfcbff5bdeac6c5e884 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 12:37:21.317184 27628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:21.316685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.1:36681)
0926 12:37:21.316958 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:21.317031 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":77,"threads_started":1}
I20260926 12:37:21.474442 27629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:21.474013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.2:37001)
0926 12:37:21.474279 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:21.474340 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":80,"threads_started":1}
I20260926 12:37:21.659848 27630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:21.659434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.3:54349)
0926 12:37:21.659698 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:21.659751 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":80,"threads_started":1}
I20260926 12:37:22.318557 27631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:22.318129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.1:34815)
0926 12:37:22.318394 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:22.318453 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":68,"threads_started":1}
I20260926 12:37:22.475757 27632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:22.475303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:45843 (local address 127.10.98.2:54649)
0926 12:37:22.475596 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:22.475659 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.62:45843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":74,"threads_started":1}
I20260926 12:37:22.644382 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 27141
I20260926 12:37:22.650007 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 27314
I20260926 12:37:22.655048 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 27448
2026-09-26T12:37:22Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59229 ms)
[----------] 30 tests from SecurityITest (245268 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:22 dist-test-slave-t7tk krb5kdc[27635](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:22 dist-test-slave-t7tk krb5kdc[27635](info): set up 2 sockets
Sep 26 12:37:22 dist-test-slave-t7tk krb5kdc[27635](info): commencing operation
krb5kdc: starting...
W20260926 12:37:23.622617 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:23 dist-test-slave-t7tk krb5kdc[27635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:37:23.745249 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:33621
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:34977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:33621
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:23.855572 27648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:23.855868 27648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:23.855929 27648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:23.860103 27648 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:23.860177 27648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:23.860203 27648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:23.860224 27648 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:23.860245 27648 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:23.865721 27648 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:34977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:33621
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:33621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:23.866779 27648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:23.867856 27648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:23.876567 27654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:23.876506 27656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:23.876698 27648 server_base.cc:1034] running on GCE node
W20260926 12:37:23.876499 27653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:23.877562 27648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:23.878700 27648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:23.879889 27648 hybrid_clock.cc:648] HybridClock initialized: now 1790426243879860 us; error 42 us; skew 500 ppm
Sep 26 12:37:23 dist-test-slave-t7tk krb5kdc[27635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426243, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:23.882831 27648 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:37:23.883868 27648 webserver.cc:460] Webserver started at http://127.10.98.62:35703/ using document root <none> and password file <none>
I20260926 12:37:23.884471 27648 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:23.884536 27648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:23.884855 27648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:23.887068 27648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "b66b13f9f1224fc3b990246a9498d91a"
format_stamp: "Formatted at 2026-09-26 12:37:23 on dist-test-slave-t7tk"
server_key: "541ebc416377b16e91b1342420d0061e"
server_key_iv: "7458156941df6caf82d761c89f127135"
server_key_version: "encryptionkey@0"
I20260926 12:37:23.887603 27648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "b66b13f9f1224fc3b990246a9498d91a"
format_stamp: "Formatted at 2026-09-26 12:37:23 on dist-test-slave-t7tk"
server_key: "541ebc416377b16e91b1342420d0061e"
server_key_iv: "7458156941df6caf82d761c89f127135"
server_key_version: "encryptionkey@0"
I20260926 12:37:23.891390 27648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:37:23.893846 27663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:23.895141 27648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:37:23.895277 27648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "b66b13f9f1224fc3b990246a9498d91a"
format_stamp: "Formatted at 2026-09-26 12:37:23 on dist-test-slave-t7tk"
server_key: "541ebc416377b16e91b1342420d0061e"
server_key_iv: "7458156941df6caf82d761c89f127135"
server_key_version: "encryptionkey@0"
I20260926 12:37:23.895378 27648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:23.922453 27648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:23.925726 27648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:23.925917 27648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:23.932617 27648 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:33621
I20260926 12:37:23.932694 27715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:33621 every 8 connection(s)
I20260926 12:37:23.933612 27648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:23.936362 27716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:23.941717 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 27648
I20260926 12:37:23.941844 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:23.942009 27716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a: Bootstrap starting.
I20260926 12:37:23.942112 10632 external_mini_cluster.cc:1428] Setting key 7e34966b495d9b44bb9b1e0e0afa2c34
I20260926 12:37:23.944504 27716 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:23.945271 27716 log.cc:824] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:23.947235 27716 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a: No bootstrap required, opened a new log
Sep 26 12:37:23 dist-test-slave-t7tk krb5kdc[27635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:37:23.951686 27716 raft_consensus.cc:348] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66b13f9f1224fc3b990246a9498d91a" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 33621 } }
I20260926 12:37:23.951946 27716 raft_consensus.cc:374] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:23.952014 27716 raft_consensus.cc:729] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b66b13f9f1224fc3b990246a9498d91a, State: Initialized, Role: FOLLOWER
I20260926 12:37:23.952600 27716 consensus_queue.cc:260] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66b13f9f1224fc3b990246a9498d91a" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 33621 } }
I20260926 12:37:23.952754 27716 raft_consensus.cc:388] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:23.952814 27716 raft_consensus.cc:482] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:23.952939 27716 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:23.953904 27716 raft_consensus.cc:504] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66b13f9f1224fc3b990246a9498d91a" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 33621 } }
I20260926 12:37:23.954257 27716 leader_election.cc:302] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b66b13f9f1224fc3b990246a9498d91a; no voters:
I20260926 12:37:23.954600 27716 leader_election.cc:288] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:23.954690 27721 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:23.954909 27721 raft_consensus.cc:686] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a [term 1 LEADER]: Becoming Leader. State: Replica: b66b13f9f1224fc3b990246a9498d91a, State: Running, Role: LEADER
I20260926 12:37:23.955271 27721 consensus_queue.cc:237] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66b13f9f1224fc3b990246a9498d91a" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 33621 } }
I20260926 12:37:23.956051 27716 sys_catalog.cc:564] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:23.957157 27722 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b66b13f9f1224fc3b990246a9498d91a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66b13f9f1224fc3b990246a9498d91a" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 33621 } } }
I20260926 12:37:23.957612 27722 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:23.957494 27723 sys_catalog.cc:455] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b66b13f9f1224fc3b990246a9498d91a. Latest consensus state: current_term: 1 leader_uuid: "b66b13f9f1224fc3b990246a9498d91a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b66b13f9f1224fc3b990246a9498d91a" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 33621 } } }
I20260926 12:37:23.957962 27723 sys_catalog.cc:458] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:23.958029 27719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:23.943516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.10.98.62:33621)
0926 12:37:23.944022 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:23.944039 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:23.944075 (+ 36us) server_negotiation.cc:407] Connection header received
0926 12:37:23.944738 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:23.944756 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:23.945091 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:23.945454 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:23.946247 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:23.947361 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:23.948586 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:23.949142 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:23.949283 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:23.952158 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:23.952165 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:23.952184 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:23.952219 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:23.954306 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:23.954827 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:23.954834 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:23.954841 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:23.955023 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:23.955361 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:23.955365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:23.955366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:23.955929 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:23.956127 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:23.956428 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 12:37:23.956760 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":231,"threads_started":1}
I20260926 12:37:23.958266 27733 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:23.961619 27733 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:23.966769 27733 catalog_manager.cc:1347] Generated new cluster ID: e2854a1f2fcc481bb3e03586f2a551ff
I20260926 12:37:23.966905 27733 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:23.982628 27733 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:23.984169 27733 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:23.991333 27733 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a: Generated new TSK 0
I20260926 12:37:23.992038 27733 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
I20260926 12:37:24.023658 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:33621
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:34977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:24.158658 27742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:24.158931 27742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:24.158981 27742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:24.163058 27742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:24.163117 27742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:24.163180 27742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:37:24.168567 27742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:34977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:33621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:24.169668 27742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:24.170696 27742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:24.180433 27748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:24.180536 27750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:24.180577 27747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:24.180644 27742 server_base.cc:1034] running on GCE node
I20260926 12:37:24.181139 27742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:24.181802 27742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:24.182977 27742 hybrid_clock.cc:648] HybridClock initialized: now 1790426244182947 us; error 47 us; skew 500 ppm
Sep 26 12:37:24 dist-test-slave-t7tk krb5kdc[27635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426244, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:24.185834 27742 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:37:24.186690 27742 webserver.cc:460] Webserver started at http://127.10.98.1:44157/ using document root <none> and password file <none>
I20260926 12:37:24.187201 27742 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:24.187256 27742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:24.187425 27742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:24.189471 27742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "629d770d34044072949f1505dab1bb1f"
format_stamp: "Formatted at 2026-09-26 12:37:24 on dist-test-slave-t7tk"
server_key: "2299b3fc37849162d5d6f39bd8df3042"
server_key_iv: "fdc3178e815ee0b3e52e1958b3a65c3c"
server_key_version: "encryptionkey@0"
I20260926 12:37:24.190042 27742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "629d770d34044072949f1505dab1bb1f"
format_stamp: "Formatted at 2026-09-26 12:37:24 on dist-test-slave-t7tk"
server_key: "2299b3fc37849162d5d6f39bd8df3042"
server_key_iv: "fdc3178e815ee0b3e52e1958b3a65c3c"
server_key_version: "encryptionkey@0"
I20260926 12:37:24.193811 27742 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 12:37:24.196291 27757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:24.197686 27742 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:37:24.197795 27742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "629d770d34044072949f1505dab1bb1f"
format_stamp: "Formatted at 2026-09-26 12:37:24 on dist-test-slave-t7tk"
server_key: "2299b3fc37849162d5d6f39bd8df3042"
server_key_iv: "fdc3178e815ee0b3e52e1958b3a65c3c"
server_key_version: "encryptionkey@0"
I20260926 12:37:24.197887 27742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:24.230571 27742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:24.233840 27742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:24.234000 27742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:24.234557 27742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:37:24.235352 27742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:37:24.235406 27742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:24.235446 27742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:37:24.235476 27742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:24.243834 27742 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:41943
I20260926 12:37:24.243868 27870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:41943 every 8 connection(s)
I20260926 12:37:24.244944 27742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:37:24.250998 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 27742
I20260926 12:37:24.251147 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:37:24.251443 10632 external_mini_cluster.cc:1428] Setting key 08b399d61daebb48fffcd9b1f2f51a68
Sep 26 12:37:24 dist-test-slave-t7tk krb5kdc[27635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426244, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:37:24.260054 27719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:24.246808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:45635 (local address 127.10.98.62:33621)
0926 12:37:24.246944 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:24.246948 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:24.247808 (+ 860us) server_negotiation.cc:407] Connection header received
0926 12:37:24.249260 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:24.249264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:24.249321 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:24.249405 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:24.250673 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.251435 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.253007 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.253353 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.253469 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:24.256094 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:24.256098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:24.256100 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:24.256127 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:24.257811 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:24.258419 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:24.258423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:24.258425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:24.258529 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:24.258847 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:24.258851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:24.258852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:24.259086 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:24.259192 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:24.259719 (+ 527us) server_negotiation.cc:299] Negotiation successful
0926 12:37:24.259875 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:37:24.260823 27873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:24.247197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:33621 (local address 127.10.98.1:45635)
0926 12:37:24.247638 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:24.247681 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:24.248906 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:24.249582 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:24.249589 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:24.249953 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:24.250503 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:24.250525 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.251609 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.251611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:24.252838 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:24.252859 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.253531 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.253535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:24.253701 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:24.254295 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:37:24.254313 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:24.255893 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:24.257946 (+ 2053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:24.257951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:24.257970 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:24.258323 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:24.258617 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:24.258619 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:24.258621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:24.258764 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:24.259206 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:24.259213 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:24.259470 (+ 257us) client_negotiation.cc:769] Sending connection context
0926 12:37:24.259646 (+ 176us) client_negotiation.cc:240] Negotiation successful
0926 12:37:24.259922 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.10.98.2@KRBTEST.COM; defaulting to no policy
I20260926 12:37:24.262102 27871 heartbeater.cc:347] Connected to a master server at 127.10.98.62:33621
I20260926 12:37:24.262353 27871 heartbeater.cc:464] Registering TS with master...
I20260926 12:37:24.262825 27871 heartbeater.cc:511] Master 127.10.98.62:33621 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.2@KRBTEST.COM" created.
I20260926 12:37:24.264595 27679 ts_manager.cc:194] Registered new tserver with Master: 629d770d34044072949f1505dab1bb1f (127.10.98.1:41943)
I20260926 12:37:24.266002 27679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:45635
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
I20260926 12:37:24.279855 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:33621
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:34977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:24.389068 27876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:24.389390 27876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:24.389448 27876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:24.393522 27876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:24.393580 27876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:24.393637 27876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:37:24.398864 27876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:34977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:33621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27876
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:24.399902 27876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:24.400909 27876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:24.410688 27884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:24.410702 27881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:24.410706 27882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:24.410885 27876 server_base.cc:1034] running on GCE node
I20260926 12:37:24.411399 27876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:24.411994 27876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:24.413175 27876 hybrid_clock.cc:648] HybridClock initialized: now 1790426244413142 us; error 47 us; skew 500 ppm
Sep 26 12:37:24 dist-test-slave-t7tk krb5kdc[27635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426244, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:24.415978 27876 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:37:24.416971 27876 webserver.cc:460] Webserver started at http://127.10.98.2:39641/ using document root <none> and password file <none>
I20260926 12:37:24.417560 27876 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:24.417627 27876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:24.417822 27876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:24.419839 27876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "9eac6d39f6ce4fc5b5d380b3ccf72c69"
format_stamp: "Formatted at 2026-09-26 12:37:24 on dist-test-slave-t7tk"
server_key: "b4be949a86b9c1c6e37beaa0870b67cc"
server_key_iv: "974f879d3f3a31eaea919a6fe9370f42"
server_key_version: "encryptionkey@0"
I20260926 12:37:24.420362 27876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "9eac6d39f6ce4fc5b5d380b3ccf72c69"
format_stamp: "Formatted at 2026-09-26 12:37:24 on dist-test-slave-t7tk"
server_key: "b4be949a86b9c1c6e37beaa0870b67cc"
server_key_iv: "974f879d3f3a31eaea919a6fe9370f42"
server_key_version: "encryptionkey@0"
I20260926 12:37:24.424005 27876 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 12:37:24.426618 27891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:24.427832 27876 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:37:24.427955 27876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "9eac6d39f6ce4fc5b5d380b3ccf72c69"
format_stamp: "Formatted at 2026-09-26 12:37:24 on dist-test-slave-t7tk"
server_key: "b4be949a86b9c1c6e37beaa0870b67cc"
server_key_iv: "974f879d3f3a31eaea919a6fe9370f42"
server_key_version: "encryptionkey@0"
I20260926 12:37:24.428103 27876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:24.439872 27876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:24.442821 27876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:24.442966 27876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:24.443544 27876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:37:24.444306 27876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:37:24.444353 27876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:24.444391 27876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:37:24.444404 27876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:24.452797 27876 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:40519
I20260926 12:37:24.452829 28004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:40519 every 8 connection(s)
I20260926 12:37:24.453722 27876 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:37:24.456132 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 27876
I20260926 12:37:24.456262 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:37:24.456663 10632 external_mini_cluster.cc:1428] Setting key 9e94beb0ac93ebecc951c08aad214de6
Sep 26 12:37:24 dist-test-slave-t7tk krb5kdc[27635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426244, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.3@KRBTEST.COM" created.
I20260926 12:37:24.470120 27719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:24.455719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:34535 (local address 127.10.98.62:33621)
0926 12:37:24.455850 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:24.455854 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:24.457581 (+ 1727us) server_negotiation.cc:407] Connection header received
0926 12:37:24.459139 (+ 1558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:24.459142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:24.459186 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:24.459247 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:24.460642 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.461437 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.462667 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.463057 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.463172 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:24.466308 (+ 3136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:24.466313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:24.466316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:24.466348 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:24.467703 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:24.468286 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:24.468289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:24.468290 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:24.468384 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:24.468862 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:24.468865 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:24.468866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:24.469132 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:24.469235 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:24.469874 (+ 639us) server_negotiation.cc:299] Negotiation successful
0926 12:37:24.469991 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:37:24.471289 28007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:24.456181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:33621 (local address 127.10.98.2:34535)
0926 12:37:24.457406 (+ 1225us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:24.457465 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:24.458891 (+ 1426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:24.459436 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:24.459445 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:24.459947 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:24.460484 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:24.460498 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.461583 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.461586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:24.462549 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:24.462559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.463227 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.463231 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:24.463386 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:24.464118 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:37:24.464140 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:24.466116 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:24.467826 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:24.467831 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:24.467845 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:24.468138 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:24.468510 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:24.468513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:24.468515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:24.468665 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:24.469248 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:24.469254 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:24.469562 (+ 308us) client_negotiation.cc:769] Sending connection context
0926 12:37:24.469795 (+ 233us) client_negotiation.cc:240] Negotiation successful
0926 12:37:24.470180 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"thread_start_us":149,"threads_started":1}
I20260926 12:37:24.472651 28005 heartbeater.cc:347] Connected to a master server at 127.10.98.62:33621
I20260926 12:37:24.472894 28005 heartbeater.cc:464] Registering TS with master...
I20260926 12:37:24.473491 28005 heartbeater.cc:511] Master 127.10.98.62:33621 requested a full tablet report, sending...
I20260926 12:37:24.474633 27679 ts_manager.cc:194] Registered new tserver with Master: 9eac6d39f6ce4fc5b5d380b3ccf72c69 (127.10.98.2:40519)
I20260926 12:37:24.475190 27679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:34535
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
I20260926 12:37:24.488916 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:33621
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:34977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:24.598556 28010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:24.598831 28010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:24.598879 28010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:24.603078 28010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:24.603147 28010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:24.603206 28010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:37:24.608552 28010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:34977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:33621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:24.609614 28010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:24.610673 28010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:37:24.620498 28010 server_base.cc:1034] running on GCE node
W20260926 12:37:24.620451 28018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:24.620457 28016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:24.620623 28015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:24.621213 28010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:24.621819 28010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:24.622980 28010 hybrid_clock.cc:648] HybridClock initialized: now 1790426244622944 us; error 48 us; skew 500 ppm
Sep 26 12:37:24 dist-test-slave-t7tk krb5kdc[27635](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426244, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:24.625643 28010 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:37:24.626484 28010 webserver.cc:460] Webserver started at http://127.10.98.3:33659/ using document root <none> and password file <none>
I20260926 12:37:24.627027 28010 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:24.627091 28010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:24.627272 28010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:24.629317 28010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "f1d3c13c1f7245989df7a78b0fdc9554"
format_stamp: "Formatted at 2026-09-26 12:37:24 on dist-test-slave-t7tk"
server_key: "319f92d19f8327871b924bc2d5b743b7"
server_key_iv: "7fd11dd7bab6c694f156445ac7006eff"
server_key_version: "encryptionkey@0"
I20260926 12:37:24.629863 28010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "f1d3c13c1f7245989df7a78b0fdc9554"
format_stamp: "Formatted at 2026-09-26 12:37:24 on dist-test-slave-t7tk"
server_key: "319f92d19f8327871b924bc2d5b743b7"
server_key_iv: "7fd11dd7bab6c694f156445ac7006eff"
server_key_version: "encryptionkey@0"
I20260926 12:37:24.633802 28010 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 12:37:24.636364 28025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:24.637667 28010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260926 12:37:24.637779 28010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "f1d3c13c1f7245989df7a78b0fdc9554"
format_stamp: "Formatted at 2026-09-26 12:37:24 on dist-test-slave-t7tk"
server_key: "319f92d19f8327871b924bc2d5b743b7"
server_key_iv: "7fd11dd7bab6c694f156445ac7006eff"
server_key_version: "encryptionkey@0"
I20260926 12:37:24.637887 28010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:24.659225 28010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:24.662042 28010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:24.662210 28010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:24.662869 28010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:37:24.663713 28010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:37:24.663767 28010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:24.663821 28010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:37:24.663844 28010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:24.672360 28010 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:44047
I20260926 12:37:24.672394 28138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:44047 every 8 connection(s)
I20260926 12:37:24.673286 28010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:37:24.675189 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 28010
I20260926 12:37:24.675313 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:37:24.675638 10632 external_mini_cluster.cc:1428] Setting key 1bb5b8fbb5a90dad31b861e8ff9d699d
Sep 26 12:37:24 dist-test-slave-t7tk krb5kdc[27635](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426244, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:37:24.687260 27719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:24.675069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:37739 (local address 127.10.98.62:33621)
0926 12:37:24.675186 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:24.675189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:24.676222 (+ 1033us) server_negotiation.cc:407] Connection header received
0926 12:37:24.677817 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:24.677819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:24.677870 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:24.677938 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:24.679143 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.679739 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.680876 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.681171 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.681251 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:24.683801 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:24.683804 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:24.683806 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:24.683831 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:24.685264 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:24.685779 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:24.685782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:24.685783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:24.685873 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:24.686181 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:24.686184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:24.686185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:24.686373 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:24.686461 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:24.687028 (+ 567us) server_negotiation.cc:299] Negotiation successful
0926 12:37:24.687138 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 12:37:24.688067 28141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:24.675560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:33621 (local address 127.10.98.3:37739)
0926 12:37:24.676072 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:24.676129 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:24.677613 (+ 1484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:24.678100 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:24.678108 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:24.678463 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:24.678940 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:24.678954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.679872 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.679874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:24.680759 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:24.680768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.681299 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.681302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:24.681454 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:24.682105 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:37:24.682126 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:24.683641 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:24.685377 (+ 1736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:24.685382 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:24.685395 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:24.685687 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:24.685946 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:24.685948 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:24.685950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:24.686080 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:24.686471 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:24.686477 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:24.686771 (+ 294us) client_negotiation.cc:769] Sending connection context
0926 12:37:24.686934 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 12:37:24.687212 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":125,"threads_started":1}
I20260926 12:37:24.689287 28139 heartbeater.cc:347] Connected to a master server at 127.10.98.62:33621
I20260926 12:37:24.689538 28139 heartbeater.cc:464] Registering TS with master...
I20260926 12:37:24.690055 28139 heartbeater.cc:511] Master 127.10.98.62:33621 requested a full tablet report, sending...
I20260926 12:37:24.691007 27679 ts_manager.cc:194] Registered new tserver with Master: f1d3c13c1f7245989df7a78b0fdc9554 (127.10.98.3:44047)
I20260926 12:37:24.691569 27679 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:37739
I20260926 12:37:24.701578 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:37:24.710599 27719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:24.703542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35026 (local address 127.10.98.62:33621)
0926 12:37:24.703668 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:24.703673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:24.703828 (+ 155us) server_negotiation.cc:407] Connection header received
0926 12:37:24.703952 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:24.703955 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:24.703999 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:24.704067 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:24.704775 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.705345 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.706211 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.706430 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.706524 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:24.707804 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:24.707808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:24.707811 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:24.707837 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:24.708988 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:24.709406 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:24.709410 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:24.709411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:24.709495 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:24.709740 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:24.709743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:24.709745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:24.709962 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:24.710087 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:24.710310 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 12:37:24.710439 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:37:24.713268 27679 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:37:24.716215 27679 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:37:24.730310 28154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:24.724827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46646 (local address 127.10.98.1:41943)
0926 12:37:24.725222 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:24.725236 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:24.725259 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:37:24.725332 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:24.725337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:24.725541 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:24.726094 (+ 553us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:24.726242 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.727288 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.729058 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.729673 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.729771 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:24.729932 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:24.730075 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 12:37:24.730167 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":92,"threads_started":1}
I20260926 12:37:24.730304 28150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:24.724214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:41943 (local address 127.0.0.1:46646)
0926 12:37:24.724741 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:24.724791 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:24.724927 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:24.725718 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:24.725724 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:24.725762 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:24.725931 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:24.725938 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.727411 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.727414 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:24.728954 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:24.728961 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.729783 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.729787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:24.729897 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:24.729933 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 12:37:24.730037 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 12:37:24.730156 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":112,"spinlock_wait_cycles":6400,"thread_start_us":82,"threads_started":1}
I20260926 12:37:24.730516 28153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:24.724041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:44047 (local address 127.0.0.1:42834)
0926 12:37:24.724743 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:24.724793 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:24.724928 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:24.725718 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:24.725724 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:24.725753 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:24.725935 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:24.725942 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.727345 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.727347 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:24.728955 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:24.728965 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.730124 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.730128 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:24.730231 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:24.730254 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:37:24.730299 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 12:37:24.730364 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":190,"spinlock_wait_cycles":12544,"thread_start_us":81,"threads_started":1}
I20260926 12:37:24.731223 28152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:24.724175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42834 (local address 127.10.98.3:44047)
0926 12:37:24.725222 (+ 1047us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:24.725238 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:24.725259 (+ 21us) server_negotiation.cc:407] Connection header received
0926 12:37:24.725332 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:24.725337 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:24.725548 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:24.725875 (+ 327us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:24.726030 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.727204 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.729153 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.729973 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.730773 (+ 800us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:24.730920 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:24.730967 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 12:37:24.731088 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":952,"thread_start_us":172,"threads_started":1}
I20260926 12:37:24.732992 28151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:24.724270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:40519 (local address 127.0.0.1:46466)
0926 12:37:24.725661 (+ 1391us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:24.725674 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:24.727683 (+ 2009us) spinlock_profiling.cc:243] Waited 1.6 ms on lock 0x1e687c88. stack: 00007f388f848c78 00007f388f8491a9 00007f388f40b1d7 00000000005984a6 00000000005984f9 000000000059c488 00007f388f8452ab 00007f388f8449da 00007f3897fa2817 00007f3897fb507f 00007f3897fab003 00007f3893e9d267 00007f3893e9b819 00007f3893edc832 00007f3893edd68c 00007f3893ef30d3
0926 12:37:24.727703 (+ 20us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:24.728669 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:24.728672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:24.728689 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:24.728776 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:24.728781 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.730182 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.730186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:24.731631 (+ 1445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:24.731641 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.732619 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.732623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:24.732706 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:24.732726 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 12:37:24.732797 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 12:37:24.732863 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1338,"mutex_wait_us":14,"spinlock_wait_cycles":3681024,"thread_start_us":83,"threads_started":1}
I20260926 12:37:24.733031 28155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:24.724978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.10.98.2:40519)
0926 12:37:24.725398 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:24.725410 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:24.725777 (+ 367us) server_negotiation.cc:407] Connection header received
0926 12:37:24.727862 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:24.727868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:24.728093 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:24.728234 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:24.728901 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.730050 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.731763 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.732443 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.732604 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:24.732738 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:24.732822 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 12:37:24.732906 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":127,"threads_started":1}
I20260926 12:37:24.733916 27805 tablet_service.cc:1467] Processing CreateTablet for tablet 29e586528ca94ace9f0b61aa284391c7 (DEFAULT_TABLE table=test-table [id=ddd77aedbcd946199880d41d506556e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:37:24.733916 28073 tablet_service.cc:1467] Processing CreateTablet for tablet 29e586528ca94ace9f0b61aa284391c7 (DEFAULT_TABLE table=test-table [id=ddd77aedbcd946199880d41d506556e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:37:24.735059 28073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29e586528ca94ace9f0b61aa284391c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:24.735054 27805 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29e586528ca94ace9f0b61aa284391c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:24.735634 27939 tablet_service.cc:1467] Processing CreateTablet for tablet 29e586528ca94ace9f0b61aa284391c7 (DEFAULT_TABLE table=test-table [id=ddd77aedbcd946199880d41d506556e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:37:24.736402 27939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 29e586528ca94ace9f0b61aa284391c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:24.742183 28156 tablet_bootstrap.cc:492] T 29e586528ca94ace9f0b61aa284391c7 P 9eac6d39f6ce4fc5b5d380b3ccf72c69: Bootstrap starting.
I20260926 12:37:24.743588 28158 tablet_bootstrap.cc:492] T 29e586528ca94ace9f0b61aa284391c7 P 629d770d34044072949f1505dab1bb1f: Bootstrap starting.
I20260926 12:37:24.745406 28157 tablet_bootstrap.cc:492] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554: Bootstrap starting.
I20260926 12:37:24.745505 28156 tablet_bootstrap.cc:654] T 29e586528ca94ace9f0b61aa284391c7 P 9eac6d39f6ce4fc5b5d380b3ccf72c69: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:24.745731 28158 tablet_bootstrap.cc:654] T 29e586528ca94ace9f0b61aa284391c7 P 629d770d34044072949f1505dab1bb1f: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:24.746418 28156 log.cc:824] T 29e586528ca94ace9f0b61aa284391c7 P 9eac6d39f6ce4fc5b5d380b3ccf72c69: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:24.746896 28158 log.cc:824] T 29e586528ca94ace9f0b61aa284391c7 P 629d770d34044072949f1505dab1bb1f: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:24.747228 28157 tablet_bootstrap.cc:654] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:24.748133 28157 log.cc:824] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:24.748338 28156 tablet_bootstrap.cc:492] T 29e586528ca94ace9f0b61aa284391c7 P 9eac6d39f6ce4fc5b5d380b3ccf72c69: No bootstrap required, opened a new log
I20260926 12:37:24.748520 28156 ts_tablet_manager.cc:1397] T 29e586528ca94ace9f0b61aa284391c7 P 9eac6d39f6ce4fc5b5d380b3ccf72c69: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 12:37:24.750156 28157 tablet_bootstrap.cc:492] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554: No bootstrap required, opened a new log
I20260926 12:37:24.750341 28157 ts_tablet_manager.cc:1397] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:37:24.750671 28158 tablet_bootstrap.cc:492] T 29e586528ca94ace9f0b61aa284391c7 P 629d770d34044072949f1505dab1bb1f: No bootstrap required, opened a new log
I20260926 12:37:24.750859 28158 ts_tablet_manager.cc:1397] T 29e586528ca94ace9f0b61aa284391c7 P 629d770d34044072949f1505dab1bb1f: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260926 12:37:24.751690 28156 raft_consensus.cc:348] T 29e586528ca94ace9f0b61aa284391c7 P 9eac6d39f6ce4fc5b5d380b3ccf72c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d3c13c1f7245989df7a78b0fdc9554" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 44047 } } peers { permanent_uuid: "629d770d34044072949f1505dab1bb1f" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41943 } } peers { permanent_uuid: "9eac6d39f6ce4fc5b5d380b3ccf72c69" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40519 } }
I20260926 12:37:24.751916 28156 raft_consensus.cc:374] T 29e586528ca94ace9f0b61aa284391c7 P 9eac6d39f6ce4fc5b5d380b3ccf72c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:24.751967 28156 raft_consensus.cc:729] T 29e586528ca94ace9f0b61aa284391c7 P 9eac6d39f6ce4fc5b5d380b3ccf72c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9eac6d39f6ce4fc5b5d380b3ccf72c69, State: Initialized, Role: FOLLOWER
I20260926 12:37:24.752381 28156 consensus_queue.cc:260] T 29e586528ca94ace9f0b61aa284391c7 P 9eac6d39f6ce4fc5b5d380b3ccf72c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d3c13c1f7245989df7a78b0fdc9554" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 44047 } } peers { permanent_uuid: "629d770d34044072949f1505dab1bb1f" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41943 } } peers { permanent_uuid: "9eac6d39f6ce4fc5b5d380b3ccf72c69" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40519 } }
I20260926 12:37:24.752985 28005 heartbeater.cc:503] Master 127.10.98.62:33621 was elected leader, sending a full tablet report...
I20260926 12:37:24.753369 28156 ts_tablet_manager.cc:1428] T 29e586528ca94ace9f0b61aa284391c7 P 9eac6d39f6ce4fc5b5d380b3ccf72c69: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:37:24.754079 28157 raft_consensus.cc:348] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d3c13c1f7245989df7a78b0fdc9554" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 44047 } } peers { permanent_uuid: "629d770d34044072949f1505dab1bb1f" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41943 } } peers { permanent_uuid: "9eac6d39f6ce4fc5b5d380b3ccf72c69" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40519 } }
I20260926 12:37:24.754349 28157 raft_consensus.cc:374] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:24.754402 28157 raft_consensus.cc:729] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1d3c13c1f7245989df7a78b0fdc9554, State: Initialized, Role: FOLLOWER
I20260926 12:37:24.754549 28158 raft_consensus.cc:348] T 29e586528ca94ace9f0b61aa284391c7 P 629d770d34044072949f1505dab1bb1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d3c13c1f7245989df7a78b0fdc9554" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 44047 } } peers { permanent_uuid: "629d770d34044072949f1505dab1bb1f" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41943 } } peers { permanent_uuid: "9eac6d39f6ce4fc5b5d380b3ccf72c69" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40519 } }
I20260926 12:37:24.754788 28158 raft_consensus.cc:374] T 29e586528ca94ace9f0b61aa284391c7 P 629d770d34044072949f1505dab1bb1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:24.754848 28158 raft_consensus.cc:729] T 29e586528ca94ace9f0b61aa284391c7 P 629d770d34044072949f1505dab1bb1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 629d770d34044072949f1505dab1bb1f, State: Initialized, Role: FOLLOWER
I20260926 12:37:24.754880 28157 consensus_queue.cc:260] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d3c13c1f7245989df7a78b0fdc9554" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 44047 } } peers { permanent_uuid: "629d770d34044072949f1505dab1bb1f" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41943 } } peers { permanent_uuid: "9eac6d39f6ce4fc5b5d380b3ccf72c69" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40519 } }
I20260926 12:37:24.755266 28158 consensus_queue.cc:260] T 29e586528ca94ace9f0b61aa284391c7 P 629d770d34044072949f1505dab1bb1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d3c13c1f7245989df7a78b0fdc9554" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 44047 } } peers { permanent_uuid: "629d770d34044072949f1505dab1bb1f" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41943 } } peers { permanent_uuid: "9eac6d39f6ce4fc5b5d380b3ccf72c69" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40519 } }
I20260926 12:37:24.755656 28139 heartbeater.cc:503] Master 127.10.98.62:33621 was elected leader, sending a full tablet report...
I20260926 12:37:24.756210 28158 ts_tablet_manager.cc:1428] T 29e586528ca94ace9f0b61aa284391c7 P 629d770d34044072949f1505dab1bb1f: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:37:24.756517 27871 heartbeater.cc:503] Master 127.10.98.62:33621 was elected leader, sending a full tablet report...
I20260926 12:37:24.756870 28157 ts_tablet_manager.cc:1428] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:37:24.797175 28163 raft_consensus.cc:482] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:37:24.797386 28163 raft_consensus.cc:504] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d3c13c1f7245989df7a78b0fdc9554" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 44047 } } peers { permanent_uuid: "629d770d34044072949f1505dab1bb1f" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41943 } } peers { permanent_uuid: "9eac6d39f6ce4fc5b5d380b3ccf72c69" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40519 } }
I20260926 12:37:24.798599 28163 leader_election.cc:288] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 629d770d34044072949f1505dab1bb1f (127.10.98.1:41943), 9eac6d39f6ce4fc5b5d380b3ccf72c69 (127.10.98.2:40519)
I20260926 12:37:24.802436 28141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:24.798869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:40519 (local address 127.10.98.3:47131)
0926 12:37:24.798976 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:24.798992 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:24.799087 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:24.799483 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:24.799486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:24.799506 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:24.799610 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:24.799615 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.800414 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.800417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:24.801335 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:24.801345 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.802048 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.802051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:24.802150 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:24.802168 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 12:37:24.802232 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 12:37:24.802303 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 12:37:24.802479 28155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:24.799082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:47131 (local address 127.10.98.2:40519)
0926 12:37:24.799227 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:24.799232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:24.799247 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:37:24.799326 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:24.799329 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:24.799382 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:24.799463 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:24.799716 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.800314 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.801452 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.801929 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.802042 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:24.802104 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:24.802309 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 12:37:24.802380 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:37:24.802677 28154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:24.798996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:34905 (local address 127.10.98.1:41943)
0926 12:37:24.799122 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:24.799127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:24.799584 (+ 457us) server_negotiation.cc:407] Connection header received
0926 12:37:24.799715 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:24.799718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:24.799759 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:24.799826 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:24.800092 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.800614 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.801685 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.802166 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.802270 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:24.802328 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:24.802482 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 12:37:24.802560 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:37:24.803150 28165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:24.798869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:41943 (local address 127.10.98.3:34905)
0926 12:37:24.799482 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:24.799497 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:24.799613 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:24.799840 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:24.799843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:24.799859 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:24.799973 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:24.799979 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.800711 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.800714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:24.801578 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:24.801587 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.802273 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.802276 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:24.802369 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:24.802392 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:37:24.802458 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 12:37:24.802915 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":285,"thread_start_us":122,"threads_started":1}
I20260926 12:37:24.803192 27959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e586528ca94ace9f0b61aa284391c7" candidate_uuid: "f1d3c13c1f7245989df7a78b0fdc9554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eac6d39f6ce4fc5b5d380b3ccf72c69" is_pre_election: true
I20260926 12:37:24.803514 27959 raft_consensus.cc:2454] T 29e586528ca94ace9f0b61aa284391c7 P 9eac6d39f6ce4fc5b5d380b3ccf72c69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1d3c13c1f7245989df7a78b0fdc9554 in term 0.
I20260926 12:37:24.804003 28026 leader_election.cc:302] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9eac6d39f6ce4fc5b5d380b3ccf72c69, f1d3c13c1f7245989df7a78b0fdc9554; no voters:
I20260926 12:37:24.804180 27825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e586528ca94ace9f0b61aa284391c7" candidate_uuid: "f1d3c13c1f7245989df7a78b0fdc9554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "629d770d34044072949f1505dab1bb1f" is_pre_election: true
I20260926 12:37:24.804317 28163 raft_consensus.cc:2781] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:37:24.804425 28163 raft_consensus.cc:482] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:37:24.804490 28163 raft_consensus.cc:3037] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:24.804477 27825 raft_consensus.cc:2454] T 29e586528ca94ace9f0b61aa284391c7 P 629d770d34044072949f1505dab1bb1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f1d3c13c1f7245989df7a78b0fdc9554 in term 0.
I20260926 12:37:24.805589 28163 raft_consensus.cc:504] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d3c13c1f7245989df7a78b0fdc9554" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 44047 } } peers { permanent_uuid: "629d770d34044072949f1505dab1bb1f" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41943 } } peers { permanent_uuid: "9eac6d39f6ce4fc5b5d380b3ccf72c69" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40519 } }
I20260926 12:37:24.805881 28163 leader_election.cc:288] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 [CANDIDATE]: Term 1 election: Requested vote from peers 629d770d34044072949f1505dab1bb1f (127.10.98.1:41943), 9eac6d39f6ce4fc5b5d380b3ccf72c69 (127.10.98.2:40519)
I20260926 12:37:24.806218 27825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e586528ca94ace9f0b61aa284391c7" candidate_uuid: "f1d3c13c1f7245989df7a78b0fdc9554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "629d770d34044072949f1505dab1bb1f"
I20260926 12:37:24.806202 27959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "29e586528ca94ace9f0b61aa284391c7" candidate_uuid: "f1d3c13c1f7245989df7a78b0fdc9554" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9eac6d39f6ce4fc5b5d380b3ccf72c69"
I20260926 12:37:24.806324 27825 raft_consensus.cc:3037] T 29e586528ca94ace9f0b61aa284391c7 P 629d770d34044072949f1505dab1bb1f [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:24.806331 27959 raft_consensus.cc:3037] T 29e586528ca94ace9f0b61aa284391c7 P 9eac6d39f6ce4fc5b5d380b3ccf72c69 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:24.807329 27825 raft_consensus.cc:2454] T 29e586528ca94ace9f0b61aa284391c7 P 629d770d34044072949f1505dab1bb1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1d3c13c1f7245989df7a78b0fdc9554 in term 1.
I20260926 12:37:24.807538 27959 raft_consensus.cc:2454] T 29e586528ca94ace9f0b61aa284391c7 P 9eac6d39f6ce4fc5b5d380b3ccf72c69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f1d3c13c1f7245989df7a78b0fdc9554 in term 1.
I20260926 12:37:24.807626 28029 leader_election.cc:302] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 629d770d34044072949f1505dab1bb1f, f1d3c13c1f7245989df7a78b0fdc9554; no voters:
I20260926 12:37:24.807783 28163 raft_consensus.cc:2781] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:24.808014 28163 raft_consensus.cc:686] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 [term 1 LEADER]: Becoming Leader. State: Replica: f1d3c13c1f7245989df7a78b0fdc9554, State: Running, Role: LEADER
I20260926 12:37:24.808312 28163 consensus_queue.cc:237] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d3c13c1f7245989df7a78b0fdc9554" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 44047 } } peers { permanent_uuid: "629d770d34044072949f1505dab1bb1f" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41943 } } peers { permanent_uuid: "9eac6d39f6ce4fc5b5d380b3ccf72c69" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40519 } }
I20260926 12:37:24.811302 27679 catalog_manager.cc:5495] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 reported cstate change: term changed from 0 to 1, leader changed from <none> to f1d3c13c1f7245989df7a78b0fdc9554 (127.10.98.3). New cstate: current_term: 1 leader_uuid: "f1d3c13c1f7245989df7a78b0fdc9554" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f1d3c13c1f7245989df7a78b0fdc9554" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 44047 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "629d770d34044072949f1505dab1bb1f" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9eac6d39f6ce4fc5b5d380b3ccf72c69" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40519 } health_report { overall_health: UNKNOWN } } }
I20260926 12:37:24.831574 28152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:24.827870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42838 (local address 127.10.98.3:44047)
0926 12:37:24.827989 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:24.827995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:24.828009 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:37:24.828128 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:24.828130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:24.828184 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:24.828269 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:37:24.828577 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.829058 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.830005 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:24.830220 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:24.830305 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:24.830645 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:37:24.831107 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:37:24.831213 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:24.831286 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 12:37:24.831425 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:37:24.837461 27825 raft_consensus.cc:1261] T 29e586528ca94ace9f0b61aa284391c7 P 629d770d34044072949f1505dab1bb1f [term 1 FOLLOWER]: Refusing update from remote peer f1d3c13c1f7245989df7a78b0fdc9554: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:37:24.837451 27959 raft_consensus.cc:1261] T 29e586528ca94ace9f0b61aa284391c7 P 9eac6d39f6ce4fc5b5d380b3ccf72c69 [term 1 FOLLOWER]: Refusing update from remote peer f1d3c13c1f7245989df7a78b0fdc9554: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:37:24.838131 28163 consensus_queue.cc:1035] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9eac6d39f6ce4fc5b5d380b3ccf72c69" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:37:24.838361 28166 consensus_queue.cc:1035] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 [LEADER]: Connected to new peer: Peer: permanent_uuid: "629d770d34044072949f1505dab1bb1f" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 41943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:37:24.846665 28171 mvcc.cc:205] Tried to move back new op lower bound from 7333585898846998528 to 7333585898737266688. Current Snapshot: MvccSnapshot[applied={T|T < 7333585898846998528}]
I20260926 12:37:24.854357 27678 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:37:24.854559 27678 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:37:24.857089 27678 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b66b13f9f1224fc3b990246a9498d91a: Sending DeleteTablet for 3 replicas of tablet 29e586528ca94ace9f0b61aa284391c7
I20260926 12:37:24.858050 28073 tablet_service.cc:1514] Processing DeleteTablet for tablet 29e586528ca94ace9f0b61aa284391c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:37:24 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:42834
I20260926 12:37:24.858079 27805 tablet_service.cc:1514] Processing DeleteTablet for tablet 29e586528ca94ace9f0b61aa284391c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:37:24 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:46646
I20260926 12:37:24.858201 27939 tablet_service.cc:1514] Processing DeleteTablet for tablet 29e586528ca94ace9f0b61aa284391c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:37:24 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:46466
I20260926 12:37:24.859001 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 27742
W20260926 12:37:24.865283 27667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.98.1:41943 (error 108)
I20260926 12:37:24.865530 28179 tablet_replica.cc:330] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554: stopping tablet replica
I20260926 12:37:24.865687 28181 tablet_replica.cc:330] T 29e586528ca94ace9f0b61aa284391c7 P 9eac6d39f6ce4fc5b5d380b3ccf72c69: stopping tablet replica
I20260926 12:37:24.865876 28179 raft_consensus.cc:2229] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:37:24.866206 28181 raft_consensus.cc:2229] T 29e586528ca94ace9f0b61aa284391c7 P 9eac6d39f6ce4fc5b5d380b3ccf72c69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:37:24.866313 28179 raft_consensus.cc:2258] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:37:24.866564 28151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:24.866276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:41943 (local address 127.0.0.1:46650)
0926 12:37:24.866397 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:24.866463 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.1:41943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 12:37:24.866580 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 27876
W20260926 12:37:24.866734 27667 catalog_manager.cc:4584] TS 629d770d34044072949f1505dab1bb1f (127.10.98.1:41943): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 29e586528ca94ace9f0b61aa284391c7: Network error: Client connection negotiation failed: client connection to 127.10.98.1:41943: connect: Connection refused (error 111)
I20260926 12:37:24.869885 28179 ts_tablet_manager.cc:1905] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:37:24.872105 28179 ts_tablet_manager.cc:1918] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:37:24.872200 28179 log.cc:1196] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/29e586528ca94ace9f0b61aa284391c7
I20260926 12:37:24.872473 28179 ts_tablet_manager.cc:1939] T 29e586528ca94ace9f0b61aa284391c7 P f1d3c13c1f7245989df7a78b0fdc9554: Deleting consensus metadata
I20260926 12:37:24.873664 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 28010
I20260926 12:37:24.874353 28151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:24.873860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:40519 (local address 127.0.0.1:46468)
0926 12:37:24.874218 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:24.874268 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.2:40519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286}
I20260926 12:37:24.878669 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 27648
2026-09-26T12:37:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2223 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T12:37:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:24Z Disabled control of system clock
I20260926 12:37:24.907133 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:34895
--webserver_interface=127.10.98.62
--webserver_port=0
--builtin_ntp_servers=127.10.98.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:34895
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 12:37:25.014110 28186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:25.014386 28186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:25.014421 28186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:25.018460 28186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:25.018525 28186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:25.018540 28186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:25.018555 28186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:25.018569 28186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:25.023418 28186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:34895
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:34895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:25.024497 28186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:25.025898 28186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:25.034049 28192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:25.034035 28191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:25.034215 28186 server_base.cc:1034] running on GCE node
W20260926 12:37:25.034039 28194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:25.034931 28186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:25.035984 28186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:25.037166 28186 hybrid_clock.cc:648] HybridClock initialized: now 1790426245037137 us; error 42 us; skew 500 ppm
I20260926 12:37:25.038949 28186 webserver.cc:460] Webserver started at http://127.10.98.62:34243/ using document root <none> and password file <none>
I20260926 12:37:25.039492 28186 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:25.039559 28186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:25.039760 28186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:25.041817 28186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "394f5f711e6a44b9b429a8ce622dc402"
format_stamp: "Formatted at 2026-09-26 12:37:25 on dist-test-slave-t7tk"
server_key: "f1b6f369a0ed6de622813cfa9a607800"
server_key_iv: "d00ce35d4cbfe721928f362c765bbba7"
server_key_version: "encryptionkey@0"
I20260926 12:37:25.042356 28186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "394f5f711e6a44b9b429a8ce622dc402"
format_stamp: "Formatted at 2026-09-26 12:37:25 on dist-test-slave-t7tk"
server_key: "f1b6f369a0ed6de622813cfa9a607800"
server_key_iv: "d00ce35d4cbfe721928f362c765bbba7"
server_key_version: "encryptionkey@0"
I20260926 12:37:25.046123 28186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:37:25.048503 28200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:25.049731 28186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:37:25.049841 28186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "394f5f711e6a44b9b429a8ce622dc402"
format_stamp: "Formatted at 2026-09-26 12:37:25 on dist-test-slave-t7tk"
server_key: "f1b6f369a0ed6de622813cfa9a607800"
server_key_iv: "d00ce35d4cbfe721928f362c765bbba7"
server_key_version: "encryptionkey@0"
I20260926 12:37:25.049934 28186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:25.064145 28186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:25.064962 28186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:25.065174 28186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:25.072225 28186 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:34895
I20260926 12:37:25.072233 28252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:34895 every 8 connection(s)
I20260926 12:37:25.073199 28186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:25.076202 28253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:25.081723 28253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402: Bootstrap starting.
I20260926 12:37:25.082538 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 28186
I20260926 12:37:25.082660 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:25.082965 10632 external_mini_cluster.cc:1428] Setting key db9cd9438ac747cc08ab16d0b04a522a
I20260926 12:37:25.084612 28253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:25.085361 28253 log.cc:824] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:25.087218 28253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402: No bootstrap required, opened a new log
I20260926 12:37:25.090787 28253 raft_consensus.cc:348] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f5f711e6a44b9b429a8ce622dc402" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 34895 } }
I20260926 12:37:25.091058 28253 raft_consensus.cc:374] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:25.091125 28253 raft_consensus.cc:729] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 394f5f711e6a44b9b429a8ce622dc402, State: Initialized, Role: FOLLOWER
I20260926 12:37:25.091538 28253 consensus_queue.cc:260] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f5f711e6a44b9b429a8ce622dc402" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 34895 } }
I20260926 12:37:25.091660 28253 raft_consensus.cc:388] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:25.091718 28253 raft_consensus.cc:482] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:25.091806 28253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:25.091971 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:25.084147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59196 (local address 127.10.98.62:34895)
0926 12:37:25.084805 (+ 658us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:25.084820 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:25.084856 (+ 36us) server_negotiation.cc:407] Connection header received
0926 12:37:25.085542 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:25.085561 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:25.085969 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:25.086289 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:25.086717 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.087699 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.088592 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.089073 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.089250 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:25.090082 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:25.090087 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:25.090101 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:25.090504 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:25.090570 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:25.090590 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:25.090676 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:25.090813 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 12:37:25.091074 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":130,"threads_started":1}
I20260926 12:37:25.092700 28253 raft_consensus.cc:504] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f5f711e6a44b9b429a8ce622dc402" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 34895 } }
I20260926 12:37:25.093016 28253 leader_election.cc:302] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 394f5f711e6a44b9b429a8ce622dc402; no voters:
I20260926 12:37:25.093348 28253 leader_election.cc:288] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:25.093432 28258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:25.093632 28258 raft_consensus.cc:686] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402 [term 1 LEADER]: Becoming Leader. State: Replica: 394f5f711e6a44b9b429a8ce622dc402, State: Running, Role: LEADER
I20260926 12:37:25.093935 28258 consensus_queue.cc:237] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f5f711e6a44b9b429a8ce622dc402" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 34895 } }
I20260926 12:37:25.094650 28253 sys_catalog.cc:564] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:25.095474 28260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 394f5f711e6a44b9b429a8ce622dc402. Latest consensus state: current_term: 1 leader_uuid: "394f5f711e6a44b9b429a8ce622dc402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f5f711e6a44b9b429a8ce622dc402" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 34895 } } }
I20260926 12:37:25.095592 28260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:25.095934 28259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "394f5f711e6a44b9b429a8ce622dc402" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "394f5f711e6a44b9b429a8ce622dc402" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 34895 } } }
I20260926 12:37:25.095979 28267 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:25.096071 28259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:25.099211 28267 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:25.104490 28267 catalog_manager.cc:1347] Generated new cluster ID: 13d9e9275c0a4ab085f47c63ebcd7756
I20260926 12:37:25.104559 28267 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:25.117568 28267 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:25.118491 28267 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:25.124801 28267 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402: Generated new TSK 0
I20260926 12:37:25.125568 28267 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:37:25.135828 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:34895
--builtin_ntp_servers=127.10.98.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 12:37:25.251171 28277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:25.251428 28277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:25.251466 28277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:25.255465 28277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:25.255522 28277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:25.255578 28277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:37:25.260346 28277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:34895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:25.261421 28277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:25.262750 28277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:25.272001 28282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:25.271961 28285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:25.272161 28277 server_base.cc:1034] running on GCE node
W20260926 12:37:25.271962 28283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:25.272675 28277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:25.273309 28277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:25.274457 28277 hybrid_clock.cc:648] HybridClock initialized: now 1790426245274441 us; error 32 us; skew 500 ppm
I20260926 12:37:25.276168 28277 webserver.cc:460] Webserver started at http://127.10.98.1:41207/ using document root <none> and password file <none>
I20260926 12:37:25.276736 28277 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:25.276803 28277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:25.276986 28277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:25.278990 28277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "1759c5d21ebb4f7da894c4974963d298"
format_stamp: "Formatted at 2026-09-26 12:37:25 on dist-test-slave-t7tk"
server_key: "de913c782dc86135b76db303082975c4"
server_key_iv: "708df4af95c068414f0344e1cb4ef65d"
server_key_version: "encryptionkey@0"
I20260926 12:37:25.279528 28277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "1759c5d21ebb4f7da894c4974963d298"
format_stamp: "Formatted at 2026-09-26 12:37:25 on dist-test-slave-t7tk"
server_key: "de913c782dc86135b76db303082975c4"
server_key_iv: "708df4af95c068414f0344e1cb4ef65d"
server_key_version: "encryptionkey@0"
I20260926 12:37:25.283119 28277 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:37:25.285468 28291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:25.286691 28277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:37:25.286805 28277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "1759c5d21ebb4f7da894c4974963d298"
format_stamp: "Formatted at 2026-09-26 12:37:25 on dist-test-slave-t7tk"
server_key: "de913c782dc86135b76db303082975c4"
server_key_iv: "708df4af95c068414f0344e1cb4ef65d"
server_key_version: "encryptionkey@0"
I20260926 12:37:25.286901 28277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:25.299320 28277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:25.300038 28277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:25.300220 28277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:25.300808 28277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:37:25.301828 28277 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:37:25.301880 28277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:25.301934 28277 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:37:25.301962 28277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:25.310907 28277 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:43865
I20260926 12:37:25.310926 28404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:43865 every 8 connection(s)
I20260926 12:37:25.311874 28277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:37:25.312266 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 28277
I20260926 12:37:25.312388 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:37:25.312700 10632 external_mini_cluster.cc:1428] Setting key f4bb165207e24b1f9d47992922035fee
I20260926 12:37:25.314843 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:34895
--builtin_ntp_servers=127.10.98.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 12:37:25.321779 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:25.313670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:41143 (local address 127.10.98.62:34895)
0926 12:37:25.313781 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:25.313784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:25.315163 (+ 1379us) server_negotiation.cc:407] Connection header received
0926 12:37:25.316536 (+ 1373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:25.316540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:25.316595 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:25.316668 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:25.317614 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.318361 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.319716 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.320058 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.320169 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:25.321262 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:25.321267 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:25.321270 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:25.321286 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:25.321313 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:25.321323 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:25.321407 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:25.321544 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 12:37:25.321614 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 12:37:25.322605 28407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:25.314265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34895 (local address 127.10.98.1:41143)
0926 12:37:25.315005 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:25.315063 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:25.316324 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:25.316850 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:25.316857 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:25.316956 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:25.317479 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:25.317493 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.318542 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.318546 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:25.319584 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:25.319595 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.320227 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.320230 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:25.320380 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:25.320951 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:37:25.320952 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:25.320995 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:37:25.321072 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:25.321414 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:25.321421 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:25.321425 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:37:25.321515 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 12:37:25.321757 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":143,"threads_started":1}
I20260926 12:37:25.323925 28405 heartbeater.cc:347] Connected to a master server at 127.10.98.62:34895
I20260926 12:37:25.324162 28405 heartbeater.cc:464] Registering TS with master...
I20260926 12:37:25.324661 28405 heartbeater.cc:511] Master 127.10.98.62:34895 requested a full tablet report, sending...
I20260926 12:37:25.326277 28217 ts_manager.cc:194] Registered new tserver with Master: 1759c5d21ebb4f7da894c4974963d298 (127.10.98.1:43865)
I20260926 12:37:25.327840 28217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.98.1:41143
W20260926 12:37:25.427426 28408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:25.427708 28408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:25.427747 28408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:25.431797 28408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:25.431855 28408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:25.431913 28408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:37:25.436817 28408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:34895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:25.437916 28408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:25.439229 28408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:25.448712 28414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:25.448769 28413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:25.448652 28416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:25.448902 28408 server_base.cc:1034] running on GCE node
I20260926 12:37:25.449352 28408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:25.449994 28408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:25.451152 28408 hybrid_clock.cc:648] HybridClock initialized: now 1790426245451133 us; error 32 us; skew 500 ppm
I20260926 12:37:25.452996 28408 webserver.cc:460] Webserver started at http://127.10.98.2:43263/ using document root <none> and password file <none>
I20260926 12:37:25.453613 28408 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:25.453686 28408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:25.453889 28408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:25.455960 28408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "348872b31c694edda133b383a0c9ecfc"
format_stamp: "Formatted at 2026-09-26 12:37:25 on dist-test-slave-t7tk"
server_key: "990f2b15b42cbeca55505fe7427cd379"
server_key_iv: "298be581fde656264b8370c93e4f6f2e"
server_key_version: "encryptionkey@0"
I20260926 12:37:25.456521 28408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "348872b31c694edda133b383a0c9ecfc"
format_stamp: "Formatted at 2026-09-26 12:37:25 on dist-test-slave-t7tk"
server_key: "990f2b15b42cbeca55505fe7427cd379"
server_key_iv: "298be581fde656264b8370c93e4f6f2e"
server_key_version: "encryptionkey@0"
I20260926 12:37:25.460240 28408 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260926 12:37:25.462800 28422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:25.464267 28408 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:37:25.464468 28408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "348872b31c694edda133b383a0c9ecfc"
format_stamp: "Formatted at 2026-09-26 12:37:25 on dist-test-slave-t7tk"
server_key: "990f2b15b42cbeca55505fe7427cd379"
server_key_iv: "298be581fde656264b8370c93e4f6f2e"
server_key_version: "encryptionkey@0"
I20260926 12:37:25.464583 28408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:25.483292 28408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:25.484083 28408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:25.484279 28408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:25.484896 28408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:37:25.485877 28408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:37:25.485937 28408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:25.485994 28408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:37:25.486018 28408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:25.494879 28408 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:40295
I20260926 12:37:25.494920 28535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:40295 every 8 connection(s)
I20260926 12:37:25.495918 28408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:37:25.502281 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 28408
I20260926 12:37:25.502403 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:37:25.502691 10632 external_mini_cluster.cc:1428] Setting key b325013f9e0694e07f7a75cd6856f953
I20260926 12:37:25.504706 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:34895
--builtin_ntp_servers=127.10.98.20:40129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 12:37:25.505719 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:25.497837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:56257 (local address 127.10.98.62:34895)
0926 12:37:25.497975 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:25.497979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:25.499158 (+ 1179us) server_negotiation.cc:407] Connection header received
0926 12:37:25.500535 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:25.500538 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:25.500586 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:25.500662 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:25.501643 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.502253 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.503539 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.503845 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.503942 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:25.505217 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:25.505220 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:25.505223 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:25.505235 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:25.505257 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:25.505265 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:25.505330 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:25.505546 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 12:37:25.505605 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:37:25.506743 28538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:25.498275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34895 (local address 127.10.98.2:56257)
0926 12:37:25.498986 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:25.499047 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:25.500328 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:25.500834 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:25.500840 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:25.500927 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:25.501498 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:25.501512 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.502370 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.502373 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:25.503434 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:25.503443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.504000 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.504004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:25.504221 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:25.504869 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:37:25.504871 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:25.504928 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:37:25.505006 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:25.505347 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:25.505356 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:25.505361 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 12:37:25.505497 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 12:37:25.505807 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":113,"threads_started":1}
I20260926 12:37:25.507863 28536 heartbeater.cc:347] Connected to a master server at 127.10.98.62:34895
I20260926 12:37:25.508124 28536 heartbeater.cc:464] Registering TS with master...
I20260926 12:37:25.508796 28536 heartbeater.cc:511] Master 127.10.98.62:34895 requested a full tablet report, sending...
I20260926 12:37:25.510025 28217 ts_manager.cc:194] Registered new tserver with Master: 348872b31c694edda133b383a0c9ecfc (127.10.98.2:40295)
I20260926 12:37:25.510802 28217 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.98.2:56257
W20260926 12:37:25.618862 28539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:25.619136 28539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:25.619174 28539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:25.623240 28539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:25.623302 28539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:25.623359 28539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:37:25.628298 28539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:40129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:34895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:25.629396 28539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:25.630736 28539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:25.640111 28547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:25.640123 28544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:25.640159 28539 server_base.cc:1034] running on GCE node
W20260926 12:37:25.640111 28545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:25.640821 28539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:25.641474 28539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:25.642630 28539 hybrid_clock.cc:648] HybridClock initialized: now 1790426245642612 us; error 29 us; skew 500 ppm
I20260926 12:37:25.644454 28539 webserver.cc:460] Webserver started at http://127.10.98.3:36051/ using document root <none> and password file <none>
I20260926 12:37:25.645162 28539 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:25.645241 28539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:25.645437 28539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:25.647562 28539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "27cacd4295c94cc88f7b72bccc6cd5f4"
format_stamp: "Formatted at 2026-09-26 12:37:25 on dist-test-slave-t7tk"
server_key: "63f2867edb417643f9a07ee5790bd9d2"
server_key_iv: "2915faf4b06b6f9c5e81dce87b2f1bba"
server_key_version: "encryptionkey@0"
I20260926 12:37:25.648113 28539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "27cacd4295c94cc88f7b72bccc6cd5f4"
format_stamp: "Formatted at 2026-09-26 12:37:25 on dist-test-slave-t7tk"
server_key: "63f2867edb417643f9a07ee5790bd9d2"
server_key_iv: "2915faf4b06b6f9c5e81dce87b2f1bba"
server_key_version: "encryptionkey@0"
I20260926 12:37:25.651868 28539 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:37:25.654650 28553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:25.655949 28539 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:37:25.656102 28539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "27cacd4295c94cc88f7b72bccc6cd5f4"
format_stamp: "Formatted at 2026-09-26 12:37:25 on dist-test-slave-t7tk"
server_key: "63f2867edb417643f9a07ee5790bd9d2"
server_key_iv: "2915faf4b06b6f9c5e81dce87b2f1bba"
server_key_version: "encryptionkey@0"
I20260926 12:37:25.656215 28539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:25.670290 28539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:25.670992 28539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:25.671169 28539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:25.671761 28539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:37:25.672710 28539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:37:25.672757 28539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:25.672796 28539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:37:25.672825 28539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:25.681212 28539 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:38321
I20260926 12:37:25.681249 28666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:38321 every 8 connection(s)
I20260926 12:37:25.682199 28539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:37:25.691413 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 28539
I20260926 12:37:25.691521 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:37:25.691588 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:25.684041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:54311 (local address 127.10.98.62:34895)
0926 12:37:25.684185 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:25.684190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:25.685390 (+ 1200us) server_negotiation.cc:407] Connection header received
0926 12:37:25.686794 (+ 1404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:25.686796 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:25.686844 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:25.686921 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:25.687884 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.688418 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.689576 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.689797 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.689878 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:25.691049 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:25.691054 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:25.691057 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:25.691071 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:25.691113 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:25.691125 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:25.691196 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:25.691359 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 12:37:25.691426 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:37:25.691826 10632 external_mini_cluster.cc:1428] Setting key 49d8ac54f16b5c69d38a54cf5321f3f8
I20260926 12:37:25.692638 28669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:25.684464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:34895 (local address 127.10.98.3:54311)
0926 12:37:25.685248 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:25.685292 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:25.686601 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:25.687094 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:25.687101 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:25.687191 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:25.687754 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:25.687767 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.688523 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.688526 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:25.689469 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:25.689478 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.689922 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.689925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:25.690118 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:25.690736 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:37:25.690738 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:25.690783 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:37:25.690860 (+ 77us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:25.691205 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:25.691215 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:25.691221 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 12:37:25.691349 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 12:37:25.691693 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":116,"threads_started":1}
I20260926 12:37:25.693699 28667 heartbeater.cc:347] Connected to a master server at 127.10.98.62:34895
I20260926 12:37:25.693958 28667 heartbeater.cc:464] Registering TS with master...
I20260926 12:37:25.694486 28667 heartbeater.cc:511] Master 127.10.98.62:34895 requested a full tablet report, sending...
I20260926 12:37:25.695437 28216 ts_manager.cc:194] Registered new tserver with Master: 27cacd4295c94cc88f7b72bccc6cd5f4 (127.10.98.3:38321)
I20260926 12:37:25.696123 28216 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.98.3:54311
I20260926 12:37:25.706130 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:37:25.711740 28256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:25.708069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59208 (local address 127.10.98.62:34895)
0926 12:37:25.708212 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:25.708216 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:25.708418 (+ 202us) server_negotiation.cc:407] Connection header received
0926 12:37:25.708554 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:25.708557 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:25.708603 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:25.708666 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:25.708940 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.709516 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.710315 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.710542 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.710628 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:25.711320 (+ 692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:25.711324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:25.711326 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:25.711340 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:25.711361 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:25.711370 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:25.711438 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:25.711534 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 12:37:25.711592 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:37:25.714669 28216 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59208:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:37:25.717746 28216 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:37:25.732024 28678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:25.726304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:38321 (local address 127.0.0.1:34412)
0926 12:37:25.726665 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:25.726854 (+ 189us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:25.726987 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:25.727942 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:25.727947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:25.727972 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:25.728155 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:25.728162 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.729333 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.729337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:25.730417 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:25.730427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.731456 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.731460 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:25.731586 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:25.731629 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 12:37:25.731744 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 12:37:25.731868 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260926 12:37:25.732057 28680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:25.726454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34412 (local address 127.10.98.3:38321)
0926 12:37:25.727085 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:25.727098 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:25.727125 (+ 27us) server_negotiation.cc:407] Connection header received
0926 12:37:25.727207 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:25.727213 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:25.727425 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:25.727586 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:25.728279 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.729226 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.730552 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.731347 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.731453 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:25.731644 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:25.731784 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 12:37:25.731884 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":130,"threads_started":1}
I20260926 12:37:25.732614 28681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:25.726405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:43865 (local address 127.0.0.1:60816)
0926 12:37:25.727318 (+ 913us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:25.727331 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:25.727411 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:25.728423 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:25.728425 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:25.728439 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:25.728517 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:25.728530 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.730030 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.730034 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:25.731117 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:25.731127 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.732256 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.732260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:25.732358 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:25.732379 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:37:25.732432 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 12:37:25.732492 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"thread_start_us":118,"threads_started":1}
I20260926 12:37:25.732746 28679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:25.727061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:40295 (local address 127.0.0.1:36248)
0926 12:37:25.727822 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:25.727835 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:25.727926 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:25.728441 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:25.728444 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:25.728461 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:25.728576 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:25.728582 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.730198 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.730201 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:25.731163 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:25.731172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.732378 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.732382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:25.732467 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:25.732490 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:37:25.732557 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 12:37:25.732625 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":41,"thread_start_us":108,"threads_started":1}
I20260926 12:37:25.733213 28682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:25.727121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60816 (local address 127.10.98.1:43865)
0926 12:37:25.727690 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:25.727702 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:25.727725 (+ 23us) server_negotiation.cc:407] Connection header received
0926 12:37:25.727797 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:25.727801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:25.728131 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:25.728286 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:25.728637 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.729902 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.731241 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.731940 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.732046 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:25.732889 (+ 843us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:25.732938 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 12:37:25.733037 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":141,"threads_started":1}
I20260926 12:37:25.733747 28683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:25.727468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36248 (local address 127.10.98.2:40295)
0926 12:37:25.727926 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:25.727937 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:25.727958 (+ 21us) server_negotiation.cc:407] Connection header received
0926 12:37:25.728029 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:25.728033 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:25.728170 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:25.728296 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:25.728693 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.729902 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.731282 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:25.732037 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:25.732135 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:25.732307 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:25.732957 (+ 650us) server_negotiation.cc:299] Negotiation successful
0926 12:37:25.733602 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":108,"threads_started":1}
I20260926 12:37:25.735196 28601 tablet_service.cc:1467] Processing CreateTablet for tablet da26f58af4924a0089674d830d4e57c8 (DEFAULT_TABLE table=test-table [id=e8d2085486204461b8d35b45e621d342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:37:25.736127 28601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da26f58af4924a0089674d830d4e57c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:25.735920 28339 tablet_service.cc:1467] Processing CreateTablet for tablet da26f58af4924a0089674d830d4e57c8 (DEFAULT_TABLE table=test-table [id=e8d2085486204461b8d35b45e621d342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:37:25.736434 28470 tablet_service.cc:1467] Processing CreateTablet for tablet da26f58af4924a0089674d830d4e57c8 (DEFAULT_TABLE table=test-table [id=e8d2085486204461b8d35b45e621d342]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:37:25.736965 28339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da26f58af4924a0089674d830d4e57c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:25.737447 28470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da26f58af4924a0089674d830d4e57c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:25.742870 28684 tablet_bootstrap.cc:492] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4: Bootstrap starting.
I20260926 12:37:25.743934 28685 tablet_bootstrap.cc:492] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298: Bootstrap starting.
I20260926 12:37:25.745014 28684 tablet_bootstrap.cc:654] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:25.745036 28686 tablet_bootstrap.cc:492] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc: Bootstrap starting.
I20260926 12:37:25.745884 28684 log.cc:824] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:25.746272 28685 tablet_bootstrap.cc:654] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:25.746812 28686 tablet_bootstrap.cc:654] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:25.747035 28685 log.cc:824] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:25.747478 28686 log.cc:824] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:25.747956 28684 tablet_bootstrap.cc:492] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4: No bootstrap required, opened a new log
I20260926 12:37:25.748136 28684 ts_tablet_manager.cc:1397] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:37:25.749015 28685 tablet_bootstrap.cc:492] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298: No bootstrap required, opened a new log
I20260926 12:37:25.749240 28686 tablet_bootstrap.cc:492] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc: No bootstrap required, opened a new log
I20260926 12:37:25.749239 28685 ts_tablet_manager.cc:1397] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:37:25.749396 28686 ts_tablet_manager.cc:1397] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260926 12:37:25.751364 28684 raft_consensus.cc:348] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cacd4295c94cc88f7b72bccc6cd5f4" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38321 } } peers { permanent_uuid: "1759c5d21ebb4f7da894c4974963d298" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 43865 } } peers { permanent_uuid: "348872b31c694edda133b383a0c9ecfc" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40295 } }
I20260926 12:37:25.751581 28684 raft_consensus.cc:374] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:25.751639 28684 raft_consensus.cc:729] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27cacd4295c94cc88f7b72bccc6cd5f4, State: Initialized, Role: FOLLOWER
I20260926 12:37:25.752045 28684 consensus_queue.cc:260] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cacd4295c94cc88f7b72bccc6cd5f4" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38321 } } peers { permanent_uuid: "1759c5d21ebb4f7da894c4974963d298" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 43865 } } peers { permanent_uuid: "348872b31c694edda133b383a0c9ecfc" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40295 } }
I20260926 12:37:25.752691 28667 heartbeater.cc:503] Master 127.10.98.62:34895 was elected leader, sending a full tablet report...
I20260926 12:37:25.753013 28684 ts_tablet_manager.cc:1428] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:37:25.753353 28685 raft_consensus.cc:348] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cacd4295c94cc88f7b72bccc6cd5f4" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38321 } } peers { permanent_uuid: "1759c5d21ebb4f7da894c4974963d298" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 43865 } } peers { permanent_uuid: "348872b31c694edda133b383a0c9ecfc" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40295 } }
I20260926 12:37:25.753353 28686 raft_consensus.cc:348] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cacd4295c94cc88f7b72bccc6cd5f4" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38321 } } peers { permanent_uuid: "1759c5d21ebb4f7da894c4974963d298" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 43865 } } peers { permanent_uuid: "348872b31c694edda133b383a0c9ecfc" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40295 } }
I20260926 12:37:25.753602 28685 raft_consensus.cc:374] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:25.753652 28685 raft_consensus.cc:729] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1759c5d21ebb4f7da894c4974963d298, State: Initialized, Role: FOLLOWER
I20260926 12:37:25.753657 28686 raft_consensus.cc:374] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:25.753703 28686 raft_consensus.cc:729] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 348872b31c694edda133b383a0c9ecfc, State: Initialized, Role: FOLLOWER
I20260926 12:37:25.754096 28685 consensus_queue.cc:260] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cacd4295c94cc88f7b72bccc6cd5f4" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38321 } } peers { permanent_uuid: "1759c5d21ebb4f7da894c4974963d298" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 43865 } } peers { permanent_uuid: "348872b31c694edda133b383a0c9ecfc" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40295 } }
I20260926 12:37:25.754096 28686 consensus_queue.cc:260] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cacd4295c94cc88f7b72bccc6cd5f4" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38321 } } peers { permanent_uuid: "1759c5d21ebb4f7da894c4974963d298" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 43865 } } peers { permanent_uuid: "348872b31c694edda133b383a0c9ecfc" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40295 } }
I20260926 12:37:25.754958 28405 heartbeater.cc:503] Master 127.10.98.62:34895 was elected leader, sending a full tablet report...
I20260926 12:37:25.755055 28685 ts_tablet_manager.cc:1428] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:37:25.755045 28686 ts_tablet_manager.cc:1428] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260926 12:37:25.755949 28536 heartbeater.cc:503] Master 127.10.98.62:34895 was elected leader, sending a full tablet report...
W20260926 12:37:25.813189 28406 tablet.cc:2369] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:37:25.933219 28668 tablet.cc:2369] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:37:25.997013 28537 tablet.cc:2369] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:37:26.046172 28691 raft_consensus.cc:482] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:37:26.046423 28691 raft_consensus.cc:504] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cacd4295c94cc88f7b72bccc6cd5f4" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38321 } } peers { permanent_uuid: "1759c5d21ebb4f7da894c4974963d298" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 43865 } } peers { permanent_uuid: "348872b31c694edda133b383a0c9ecfc" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40295 } }
I20260926 12:37:26.047688 28691 leader_election.cc:288] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27cacd4295c94cc88f7b72bccc6cd5f4 (127.10.98.3:38321), 348872b31c694edda133b383a0c9ecfc (127.10.98.2:40295)
I20260926 12:37:26.051635 28683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:26.048063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:49309 (local address 127.10.98.2:40295)
0926 12:37:26.048190 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:26.048195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:26.048549 (+ 354us) server_negotiation.cc:407] Connection header received
0926 12:37:26.048618 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:26.048620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:26.048669 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:26.048744 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:26.049022 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:26.049534 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:26.050717 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:26.051099 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:26.051185 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:26.051245 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:26.051425 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 12:37:26.051506 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 12:37:26.051642 28680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:26.048066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:49299 (local address 127.10.98.3:38321)
0926 12:37:26.048190 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:26.048195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:26.048375 (+ 180us) server_negotiation.cc:407] Connection header received
0926 12:37:26.048483 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:26.048486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:26.048535 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:26.048605 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:26.048872 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:26.049525 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:26.050709 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:26.051114 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:26.051205 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:26.051262 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:26.051436 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 12:37:26.051506 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:37:26.051683 28694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:26.047990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:40295 (local address 127.10.98.1:49309)
0926 12:37:26.048421 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:26.048434 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:26.048528 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:26.048760 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:26.048762 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:26.048777 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:26.048909 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:26.048916 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:26.049635 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:26.049638 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:26.050590 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:26.050599 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:26.051209 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:26.051212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:26.051308 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:26.051330 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:37:26.051391 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 12:37:26.051500 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":80,"thread_start_us":74,"threads_started":1}
I20260926 12:37:26.051689 28693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:26.047938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:38321 (local address 127.10.98.1:49299)
0926 12:37:26.048290 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:26.048302 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:26.048398 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:26.048625 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:26.048628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:26.048645 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:26.048767 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:26.048774 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:26.049626 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:26.049630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:26.050594 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:26.050603 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:26.051223 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:26.051239 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:26.051323 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:26.051339 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 12:37:26.051420 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 12:37:26.051527 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"mutex_wait_us":94,"thread_start_us":91,"threads_started":1}
I20260926 12:37:26.052292 28490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da26f58af4924a0089674d830d4e57c8" candidate_uuid: "1759c5d21ebb4f7da894c4974963d298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "348872b31c694edda133b383a0c9ecfc" is_pre_election: true
I20260926 12:37:26.052393 28621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da26f58af4924a0089674d830d4e57c8" candidate_uuid: "1759c5d21ebb4f7da894c4974963d298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27cacd4295c94cc88f7b72bccc6cd5f4" is_pre_election: true
I20260926 12:37:26.052596 28490 raft_consensus.cc:2454] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1759c5d21ebb4f7da894c4974963d298 in term 0.
I20260926 12:37:26.052659 28621 raft_consensus.cc:2454] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1759c5d21ebb4f7da894c4974963d298 in term 0.
I20260926 12:37:26.053071 28292 leader_election.cc:302] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1759c5d21ebb4f7da894c4974963d298, 348872b31c694edda133b383a0c9ecfc; no voters:
I20260926 12:37:26.053617 28691 raft_consensus.cc:2781] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:37:26.053695 28691 raft_consensus.cc:482] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:37:26.053732 28691 raft_consensus.cc:3037] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:26.054759 28691 raft_consensus.cc:504] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cacd4295c94cc88f7b72bccc6cd5f4" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38321 } } peers { permanent_uuid: "1759c5d21ebb4f7da894c4974963d298" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 43865 } } peers { permanent_uuid: "348872b31c694edda133b383a0c9ecfc" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40295 } }
I20260926 12:37:26.055095 28691 leader_election.cc:288] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 [CANDIDATE]: Term 1 election: Requested vote from peers 27cacd4295c94cc88f7b72bccc6cd5f4 (127.10.98.3:38321), 348872b31c694edda133b383a0c9ecfc (127.10.98.2:40295)
I20260926 12:37:26.055583 28490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da26f58af4924a0089674d830d4e57c8" candidate_uuid: "1759c5d21ebb4f7da894c4974963d298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "348872b31c694edda133b383a0c9ecfc"
I20260926 12:37:26.055733 28490 raft_consensus.cc:3037] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:26.056780 28490 raft_consensus.cc:2454] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1759c5d21ebb4f7da894c4974963d298 in term 1.
I20260926 12:37:26.057086 28621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da26f58af4924a0089674d830d4e57c8" candidate_uuid: "1759c5d21ebb4f7da894c4974963d298" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27cacd4295c94cc88f7b72bccc6cd5f4"
I20260926 12:37:26.057148 28292 leader_election.cc:302] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1759c5d21ebb4f7da894c4974963d298, 348872b31c694edda133b383a0c9ecfc; no voters:
I20260926 12:37:26.057258 28621 raft_consensus.cc:3037] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:26.057313 28691 raft_consensus.cc:2781] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:26.057569 28691 raft_consensus.cc:686] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 [term 1 LEADER]: Becoming Leader. State: Replica: 1759c5d21ebb4f7da894c4974963d298, State: Running, Role: LEADER
I20260926 12:37:26.057999 28691 consensus_queue.cc:237] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cacd4295c94cc88f7b72bccc6cd5f4" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38321 } } peers { permanent_uuid: "1759c5d21ebb4f7da894c4974963d298" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 43865 } } peers { permanent_uuid: "348872b31c694edda133b383a0c9ecfc" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40295 } }
I20260926 12:37:26.058715 28621 raft_consensus.cc:2454] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1759c5d21ebb4f7da894c4974963d298 in term 1.
I20260926 12:37:26.061709 28217 catalog_manager.cc:5495] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1759c5d21ebb4f7da894c4974963d298 (127.10.98.1). New cstate: current_term: 1 leader_uuid: "1759c5d21ebb4f7da894c4974963d298" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27cacd4295c94cc88f7b72bccc6cd5f4" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38321 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1759c5d21ebb4f7da894c4974963d298" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 43865 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "348872b31c694edda133b383a0c9ecfc" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40295 } health_report { overall_health: UNKNOWN } } }
I20260926 12:37:26.097445 28682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:26.093868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60820 (local address 127.10.98.1:43865)
0926 12:37:26.094001 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:26.094005 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:26.094021 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:37:26.094108 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:26.094110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:26.094166 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:26.094253 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:37:26.094590 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:26.095133 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:26.096026 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:26.096244 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:26.096331 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:26.096608 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:37:26.097070 (+ 462us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:37:26.097204 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:26.097251 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 12:37:26.097305 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:37:26.103190 28621 raft_consensus.cc:1261] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4 [term 1 FOLLOWER]: Refusing update from remote peer 1759c5d21ebb4f7da894c4974963d298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:37:26.103193 28490 raft_consensus.cc:1261] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc [term 1 FOLLOWER]: Refusing update from remote peer 1759c5d21ebb4f7da894c4974963d298: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:37:26.103838 28695 consensus_queue.cc:1035] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "348872b31c694edda133b383a0c9ecfc" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:37:26.104097 28691 consensus_queue.cc:1035] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27cacd4295c94cc88f7b72bccc6cd5f4" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:37:26.112586 28699 mvcc.cc:205] Tried to move back new op lower bound from 7333585904031739904 to 7333585903856963584. Current Snapshot: MvccSnapshot[applied={T|T < 7333585904031739904}]
I20260926 12:37:26.113451 28700 mvcc.cc:205] Tried to move back new op lower bound from 7333585904031739904 to 7333585903856963584. Current Snapshot: MvccSnapshot[applied={T|T < 7333585904031739904 or (T in {7333585904031739904})}]
I20260926 12:37:26.119246 28216 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:37:26.119398 28216 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59208:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:37:26.121646 28216 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 394f5f711e6a44b9b429a8ce622dc402: Sending DeleteTablet for 3 replicas of tablet da26f58af4924a0089674d830d4e57c8
I20260926 12:37:26.122380 28601 tablet_service.cc:1514] Processing DeleteTablet for tablet da26f58af4924a0089674d830d4e57c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:37:26 UTC) from {username='slave'} at 127.0.0.1:34412
I20260926 12:37:26.122570 28339 tablet_service.cc:1514] Processing DeleteTablet for tablet da26f58af4924a0089674d830d4e57c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:37:26 UTC) from {username='slave'} at 127.0.0.1:60816
I20260926 12:37:26.122575 28470 tablet_service.cc:1514] Processing DeleteTablet for tablet da26f58af4924a0089674d830d4e57c8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:37:26 UTC) from {username='slave'} at 127.0.0.1:36248
I20260926 12:37:26.123060 28708 tablet_replica.cc:330] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298: stopping tablet replica
I20260926 12:37:26.123149 28707 tablet_replica.cc:330] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4: stopping tablet replica
I20260926 12:37:26.123464 28708 raft_consensus.cc:2229] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:37:26.123463 28707 raft_consensus.cc:2229] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:37:26.123795 28707 raft_consensus.cc:2258] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:37:26.123808 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 28277
I20260926 12:37:26.123979 28708 raft_consensus.cc:2258] T da26f58af4924a0089674d830d4e57c8 P 1759c5d21ebb4f7da894c4974963d298 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:37:26.124570 28709 tablet_replica.cc:330] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc: stopping tablet replica
I20260926 12:37:26.124850 28709 raft_consensus.cc:2229] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:37:26.125155 28709 raft_consensus.cc:2258] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:37:26.126605 28709 ts_tablet_manager.cc:1905] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:37:26.127344 28707 ts_tablet_manager.cc:1905] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:37:26.128880 28709 ts_tablet_manager.cc:1918] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:37:26.128966 28709 log.cc:1196] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/da26f58af4924a0089674d830d4e57c8
I20260926 12:37:26.129272 28709 ts_tablet_manager.cc:1939] T da26f58af4924a0089674d830d4e57c8 P 348872b31c694edda133b383a0c9ecfc: Deleting consensus metadata
I20260926 12:37:26.129940 28707 ts_tablet_manager.cc:1918] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:37:26.130045 28707 log.cc:1196] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/da26f58af4924a0089674d830d4e57c8
I20260926 12:37:26.130306 28707 ts_tablet_manager.cc:1939] T da26f58af4924a0089674d830d4e57c8 P 27cacd4295c94cc88f7b72bccc6cd5f4: Deleting consensus metadata
I20260926 12:37:26.130520 28201 catalog_manager.cc:4841] TS 348872b31c694edda133b383a0c9ecfc (127.10.98.2:40295): tablet da26f58af4924a0089674d830d4e57c8 (table test-table [id=e8d2085486204461b8d35b45e621d342]) successfully deleted
I20260926 12:37:26.131413 28204 catalog_manager.cc:4841] TS 27cacd4295c94cc88f7b72bccc6cd5f4 (127.10.98.3:38321): tablet da26f58af4924a0089674d830d4e57c8 (table test-table [id=e8d2085486204461b8d35b45e621d342]) successfully deleted
W20260926 12:37:26.131866 28202 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.98.1:43865 (error 108)
I20260926 12:37:26.132076 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 28408
I20260926 12:37:26.133039 28679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:26.132786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:43865 (local address 127.0.0.1:60836)
0926 12:37:26.132896 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:26.132962 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.1:43865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 12:37:26.133203 28202 catalog_manager.cc:4584] TS 1759c5d21ebb4f7da894c4974963d298 (127.10.98.1:43865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet da26f58af4924a0089674d830d4e57c8: Network error: Client connection negotiation failed: client connection to 127.10.98.1:43865: connect: Connection refused (error 111)
I20260926 12:37:26.137889 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 28539
I20260926 12:37:26.142644 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 28186
2026-09-26T12:37:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1263 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:26 dist-test-slave-t7tk krb5kdc[28713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:26 dist-test-slave-t7tk krb5kdc[28713](info): set up 2 sockets
Sep 26 12:37:26 dist-test-slave-t7tk krb5kdc[28713](info): commencing operation
krb5kdc: starting...
W20260926 12:37:27.106412 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:27 dist-test-slave-t7tk krb5kdc[28713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:37:27.224538 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:41133
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:41133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:27.334227 28726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:27.334518 28726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:27.334574 28726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:27.338572 28726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:27.338634 28726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:27.338660 28726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:27.338680 28726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:27.338701 28726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:27.343991 28726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:41133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:41133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:27.345010 28726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:27.346035 28726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:37:27.354460 28726 server_base.cc:1034] running on GCE node
W20260926 12:37:27.354378 28731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:27.354378 28732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:27.354378 28734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:27.355192 28726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:27.356277 28726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:27.357458 28726 hybrid_clock.cc:648] HybridClock initialized: now 1790426247357419 us; error 50 us; skew 500 ppm
Sep 26 12:37:27 dist-test-slave-t7tk krb5kdc[28713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426247, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:27.360169 28726 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:37:27.361145 28726 webserver.cc:460] Webserver started at http://127.10.98.62:40689/ using document root <none> and password file <none>
I20260926 12:37:27.361698 28726 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:27.361758 28726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:27.361958 28726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:27.364001 28726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "a22e0a364b3d4f2a81373cb6996a651c"
format_stamp: "Formatted at 2026-09-26 12:37:27 on dist-test-slave-t7tk"
server_key: "b9425f4c103b7126cfe8b8d7a4f2978a"
server_key_iv: "728a10d0d64cd2f5e3be8d22b0c30dc7"
server_key_version: "encryptionkey@0"
I20260926 12:37:27.364544 28726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "a22e0a364b3d4f2a81373cb6996a651c"
format_stamp: "Formatted at 2026-09-26 12:37:27 on dist-test-slave-t7tk"
server_key: "b9425f4c103b7126cfe8b8d7a4f2978a"
server_key_iv: "728a10d0d64cd2f5e3be8d22b0c30dc7"
server_key_version: "encryptionkey@0"
I20260926 12:37:27.368181 28726 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 12:37:27.370631 28741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:27.371824 28726 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:37:27.371930 28726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "a22e0a364b3d4f2a81373cb6996a651c"
format_stamp: "Formatted at 2026-09-26 12:37:27 on dist-test-slave-t7tk"
server_key: "b9425f4c103b7126cfe8b8d7a4f2978a"
server_key_iv: "728a10d0d64cd2f5e3be8d22b0c30dc7"
server_key_version: "encryptionkey@0"
I20260926 12:37:27.372040 28726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:27.392019 28726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:27.394907 28726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:27.395082 28726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:27.401791 28726 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:41133
I20260926 12:37:27.401808 28793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:41133 every 8 connection(s)
I20260926 12:37:27.402729 28726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:27.405483 28794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:27.410588 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 28726
I20260926 12:37:27.410732 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:27.411011 10632 external_mini_cluster.cc:1428] Setting key 936875663a115b0ce5c292fd8ed8bda0
I20260926 12:37:27.411000 28794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c: Bootstrap starting.
I20260926 12:37:27.413461 28794 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:27.414242 28794 log.cc:824] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:27.416275 28794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c: No bootstrap required, opened a new log
Sep 26 12:37:27 dist-test-slave-t7tk krb5kdc[28713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:37:27.420434 28794 raft_consensus.cc:348] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22e0a364b3d4f2a81373cb6996a651c" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41133 } }
I20260926 12:37:27.420678 28794 raft_consensus.cc:374] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:27.420738 28794 raft_consensus.cc:729] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a22e0a364b3d4f2a81373cb6996a651c, State: Initialized, Role: FOLLOWER
I20260926 12:37:27.421242 28794 consensus_queue.cc:260] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22e0a364b3d4f2a81373cb6996a651c" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41133 } }
I20260926 12:37:27.421396 28794 raft_consensus.cc:388] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:27.421459 28794 raft_consensus.cc:482] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:27.421576 28794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:27.422773 28794 raft_consensus.cc:504] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22e0a364b3d4f2a81373cb6996a651c" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41133 } }
I20260926 12:37:27.423157 28794 leader_election.cc:302] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a22e0a364b3d4f2a81373cb6996a651c; no voters:
I20260926 12:37:27.423496 28794 leader_election.cc:288] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:27.423593 28799 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:27.423802 28799 raft_consensus.cc:686] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c [term 1 LEADER]: Becoming Leader. State: Replica: a22e0a364b3d4f2a81373cb6996a651c, State: Running, Role: LEADER
I20260926 12:37:27.424121 28799 consensus_queue.cc:237] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22e0a364b3d4f2a81373cb6996a651c" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41133 } }
I20260926 12:37:27.425045 28794 sys_catalog.cc:564] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:27.425787 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:27.412278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35464 (local address 127.10.98.62:41133)
0926 12:37:27.412663 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:27.412679 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:27.412713 (+ 34us) server_negotiation.cc:407] Connection header received
0926 12:37:27.413339 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:27.413356 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:27.413713 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:27.413994 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:27.414847 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:27.415958 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:27.417121 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:27.417588 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:27.417712 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:27.420492 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:27.420497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:27.420511 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:27.420538 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:27.422308 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:27.422835 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:27.422840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:27.422845 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:27.423002 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:27.423340 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:27.423344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:27.423346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:27.423895 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:27.424207 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:27.424439 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 12:37:27.424751 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":107,"threads_started":1}
I20260926 12:37:27.426123 28801 sys_catalog.cc:455] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a22e0a364b3d4f2a81373cb6996a651c. Latest consensus state: current_term: 1 leader_uuid: "a22e0a364b3d4f2a81373cb6996a651c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22e0a364b3d4f2a81373cb6996a651c" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41133 } } }
I20260926 12:37:27.426322 28801 sys_catalog.cc:458] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:27.426618 28800 sys_catalog.cc:455] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a22e0a364b3d4f2a81373cb6996a651c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a22e0a364b3d4f2a81373cb6996a651c" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 41133 } } }
I20260926 12:37:27.426764 28808 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:27.426772 28800 sys_catalog.cc:458] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:27.429873 28808 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:27.435580 28808 catalog_manager.cc:1347] Generated new cluster ID: 9eb1911e4dd24076801040fef05a24b7
I20260926 12:37:27.435653 28808 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:27.446882 28808 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:27.448220 28808 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:27.462373 28808 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c: Generated new TSK 0
I20260926 12:37:27.463165 28808 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
I20260926 12:37:27.496843 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:41133
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:27.637976 28820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:27.638250 28820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:27.638293 28820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:27.642236 28820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:27.642288 28820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:27.642345 28820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:37:27.647538 28820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:41133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:27.648502 28820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:27.649454 28820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:27.658823 28825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:27.658933 28826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:27.658946 28828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:27.659139 28820 server_base.cc:1034] running on GCE node
I20260926 12:37:27.659523 28820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:27.660163 28820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:27.661319 28820 hybrid_clock.cc:648] HybridClock initialized: now 1790426247661293 us; error 36 us; skew 500 ppm
Sep 26 12:37:27 dist-test-slave-t7tk krb5kdc[28713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426247, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:27.663919 28820 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:37:27.664741 28820 webserver.cc:460] Webserver started at http://127.10.98.1:33597/ using document root <none> and password file <none>
I20260926 12:37:27.665313 28820 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:27.665380 28820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:27.665623 28820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:27.667639 28820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "7c98562bcaef4057a62fe6836f04710a"
format_stamp: "Formatted at 2026-09-26 12:37:27 on dist-test-slave-t7tk"
server_key: "3804ed7eaff153c4b1b0587eb899e16c"
server_key_iv: "722bb03760235874bba2a563df06bc09"
server_key_version: "encryptionkey@0"
I20260926 12:37:27.668164 28820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "7c98562bcaef4057a62fe6836f04710a"
format_stamp: "Formatted at 2026-09-26 12:37:27 on dist-test-slave-t7tk"
server_key: "3804ed7eaff153c4b1b0587eb899e16c"
server_key_iv: "722bb03760235874bba2a563df06bc09"
server_key_version: "encryptionkey@0"
I20260926 12:37:27.671970 28820 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:37:27.674435 28835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:27.675827 28820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:37:27.675951 28820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "7c98562bcaef4057a62fe6836f04710a"
format_stamp: "Formatted at 2026-09-26 12:37:27 on dist-test-slave-t7tk"
server_key: "3804ed7eaff153c4b1b0587eb899e16c"
server_key_iv: "722bb03760235874bba2a563df06bc09"
server_key_version: "encryptionkey@0"
I20260926 12:37:27.676074 28820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:27.698796 28820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:27.701818 28820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:27.701998 28820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:27.702595 28820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:37:27.703419 28820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:37:27.703464 28820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:27.703517 28820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:37:27.703552 28820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:27.712296 28820 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:36357
I20260926 12:37:27.712327 28948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:36357 every 8 connection(s)
I20260926 12:37:27.713228 28820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:37:27.714279 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 28820
I20260926 12:37:27.714408 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:37:27.714715 10632 external_mini_cluster.cc:1428] Setting key 122ec75485db79ee9b9a725492b3cb46
WARNING: no policy specified for kudu/127.10.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.2@KRBTEST.COM" created.
Sep 26 12:37:27 dist-test-slave-t7tk krb5kdc[28713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426247, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:37:27.734248 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:27.715141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:58835 (local address 127.10.98.62:41133)
0926 12:37:27.715284 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:27.715288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:27.716679 (+ 1391us) server_negotiation.cc:407] Connection header received
0926 12:37:27.717835 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:27.717837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:27.717883 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:27.717974 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:27.719196 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:27.719758 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:27.721463 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:27.721691 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:27.721792 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:27.730890 (+ 9098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:27.730894 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:27.730896 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:27.730922 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:27.732186 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:27.732699 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:27.732702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:27.732703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:27.732777 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:27.733127 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:27.733130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:27.733131 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:27.733322 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:27.733411 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:27.733941 (+ 530us) server_negotiation.cc:299] Negotiation successful
0926 12:37:27.734080 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:37:27.735061 28951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:27.715663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:41133 (local address 127.10.98.1:58835)
0926 12:37:27.716513 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:27.716571 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:27.717621 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:27.718151 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:27.718158 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:27.718521 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:27.719036 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:27.719055 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:27.719879 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:27.719882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:27.721338 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:27.721350 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:27.721848 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:27.721853 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:27.722053 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:27.728951 (+ 6898us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:37:27.728975 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:27.730701 (+ 1726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:27.732302 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:27.732309 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:27.732326 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:27.732593 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:27.732859 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:27.732862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:27.732864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:27.733007 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:27.733423 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:27.733431 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:27.733699 (+ 268us) client_negotiation.cc:769] Sending connection context
0926 12:37:27.733867 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 12:37:27.734217 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":141,"threads_started":1}
I20260926 12:37:27.736333 28949 heartbeater.cc:347] Connected to a master server at 127.10.98.62:41133
I20260926 12:37:27.736580 28949 heartbeater.cc:464] Registering TS with master...
I20260926 12:37:27.737125 28949 heartbeater.cc:511] Master 127.10.98.62:41133 requested a full tablet report, sending...
I20260926 12:37:27.738696 28758 ts_manager.cc:194] Registered new tserver with Master: 7c98562bcaef4057a62fe6836f04710a (127.10.98.1:36357)
I20260926 12:37:27.739949 28758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:58835
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
I20260926 12:37:27.742189 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:41133
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:27.849824 28954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:27.850107 28954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:27.850152 28954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:27.854219 28954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:27.854281 28954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:27.854363 28954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:37:27.859870 28954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:41133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:27.860950 28954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:27.862046 28954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:27.871677 28959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:27.871734 28954 server_base.cc:1034] running on GCE node
W20260926 12:37:27.871697 28962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:27.871671 28960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:27.872365 28954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:27.873034 28954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:27.874197 28954 hybrid_clock.cc:648] HybridClock initialized: now 1790426247874176 us; error 34 us; skew 500 ppm
Sep 26 12:37:27 dist-test-slave-t7tk krb5kdc[28713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426247, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:27.876995 28954 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:37:27.877914 28954 webserver.cc:460] Webserver started at http://127.10.98.2:33461/ using document root <none> and password file <none>
I20260926 12:37:27.878484 28954 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:27.878547 28954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:27.878742 28954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:27.880883 28954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "c0b5972966ec4eefb08d92f0df7fdb37"
format_stamp: "Formatted at 2026-09-26 12:37:27 on dist-test-slave-t7tk"
server_key: "c7cc881644f4f485064e67f2e6f73a60"
server_key_iv: "9f08628a75b943c927176ad033e1ac50"
server_key_version: "encryptionkey@0"
I20260926 12:37:27.881500 28954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "c0b5972966ec4eefb08d92f0df7fdb37"
format_stamp: "Formatted at 2026-09-26 12:37:27 on dist-test-slave-t7tk"
server_key: "c7cc881644f4f485064e67f2e6f73a60"
server_key_iv: "9f08628a75b943c927176ad033e1ac50"
server_key_version: "encryptionkey@0"
I20260926 12:37:27.885140 28954 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:37:27.887455 28969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:27.888656 28954 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:37:27.888753 28954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "c0b5972966ec4eefb08d92f0df7fdb37"
format_stamp: "Formatted at 2026-09-26 12:37:27 on dist-test-slave-t7tk"
server_key: "c7cc881644f4f485064e67f2e6f73a60"
server_key_iv: "9f08628a75b943c927176ad033e1ac50"
server_key_version: "encryptionkey@0"
I20260926 12:37:27.888836 28954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:27.905897 28954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:27.908684 28954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:27.908835 28954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:27.909405 28954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:37:27.910193 28954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:37:27.910238 28954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:27.910279 28954 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:37:27.910298 28954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:27.918853 28954 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:37905
I20260926 12:37:27.918880 29082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:37905 every 8 connection(s)
I20260926 12:37:27.919731 28954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:37:27.928272 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 28954
I20260926 12:37:27.928398 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:37:27.928690 10632 external_mini_cluster.cc:1428] Setting key ede6a23c6ededeaf2c644dd8ccdd104a
Sep 26 12:37:27 dist-test-slave-t7tk krb5kdc[28713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426247, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:37:27.933058 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:27.921515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:49779 (local address 127.10.98.62:41133)
0926 12:37:27.921699 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:27.921703 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:27.922484 (+ 781us) server_negotiation.cc:407] Connection header received
0926 12:37:27.923689 (+ 1205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:27.923691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:27.923739 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:27.923808 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:27.924982 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:27.925637 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:27.927014 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:27.927293 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:27.927370 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:27.929902 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:27.929905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:27.929907 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:27.929929 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:27.931206 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:27.931673 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:27.931676 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:27.931677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:27.931747 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:27.932048 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:27.932052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:27.932053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:27.932221 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:27.932303 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:27.932789 (+ 486us) server_negotiation.cc:299] Negotiation successful
0926 12:37:27.932900 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 12:37:27.933959 29085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:27.921871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:41133 (local address 127.10.98.2:49779)
0926 12:37:27.922321 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:27.922365 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:27.923501 (+ 1136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:27.923943 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:27.923949 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:27.924330 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:27.924830 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:27.924849 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:27.925788 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:27.925791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:27.926902 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:27.926912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:27.927410 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:27.927412 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:27.927601 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:27.928119 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:37:27.928139 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:27.929743 (+ 1604us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:27.931313 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:27.931318 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:27.931334 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:27.931582 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:27.931828 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:27.931831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:27.931833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:27.931965 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:27.932316 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:27.932322 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:27.932566 (+ 244us) client_negotiation.cc:769] Sending connection context
0926 12:37:27.932730 (+ 164us) client_negotiation.cc:240] Negotiation successful
0926 12:37:27.933059 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":121,"threads_started":1}
I20260926 12:37:27.935047 29083 heartbeater.cc:347] Connected to a master server at 127.10.98.62:41133
I20260926 12:37:27.935264 29083 heartbeater.cc:464] Registering TS with master...
I20260926 12:37:27.935720 29083 heartbeater.cc:511] Master 127.10.98.62:41133 requested a full tablet report, sending...
I20260926 12:37:27.936707 28758 ts_manager.cc:194] Registered new tserver with Master: c0b5972966ec4eefb08d92f0df7fdb37 (127.10.98.2:37905)
WARNING: no policy specified for kudu/127.10.98.3@KRBTEST.COM; defaulting to no policy
I20260926 12:37:27.937399 28758 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:49779
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
I20260926 12:37:27.954186 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:41133
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:28.062592 29088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:28.062845 29088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:28.062887 29088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:28.066848 29088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:28.066902 29088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:28.066958 29088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:37:28.072114 29088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:41133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:28.073112 29088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:28.074090 29088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:37:28.084177 29088 server_base.cc:1034] running on GCE node
W20260926 12:37:28.084226 29093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:28.084326 29094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:28.084226 29096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:28.084769 29088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:28.085440 29088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:28.086618 29088 hybrid_clock.cc:648] HybridClock initialized: now 1790426248086598 us; error 33 us; skew 500 ppm
Sep 26 12:37:28 dist-test-slave-t7tk krb5kdc[28713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426248, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:28.089277 29088 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:37:28.090186 29088 webserver.cc:460] Webserver started at http://127.10.98.3:36855/ using document root <none> and password file <none>
I20260926 12:37:28.090747 29088 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:28.090813 29088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:28.091001 29088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:28.093159 29088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "a9a779dd515c4adba750eda2785058ff"
format_stamp: "Formatted at 2026-09-26 12:37:28 on dist-test-slave-t7tk"
server_key: "0515ef58c69e2e760f5c27fe3d9d9dc3"
server_key_iv: "87cb8759bc4119e7b20c783538633be5"
server_key_version: "encryptionkey@0"
I20260926 12:37:28.093711 29088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "a9a779dd515c4adba750eda2785058ff"
format_stamp: "Formatted at 2026-09-26 12:37:28 on dist-test-slave-t7tk"
server_key: "0515ef58c69e2e760f5c27fe3d9d9dc3"
server_key_iv: "87cb8759bc4119e7b20c783538633be5"
server_key_version: "encryptionkey@0"
I20260926 12:37:28.097330 29088 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:37:28.099732 29103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:28.100930 29088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:37:28.101035 29088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "a9a779dd515c4adba750eda2785058ff"
format_stamp: "Formatted at 2026-09-26 12:37:28 on dist-test-slave-t7tk"
server_key: "0515ef58c69e2e760f5c27fe3d9d9dc3"
server_key_iv: "87cb8759bc4119e7b20c783538633be5"
server_key_version: "encryptionkey@0"
I20260926 12:37:28.101164 29088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:28.114845 29088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:28.117753 29088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:28.117933 29088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:28.118507 29088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:37:28.119313 29088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:37:28.119360 29088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:28.119414 29088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:37:28.119442 29088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:28.127898 29088 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:38307
I20260926 12:37:28.127933 29216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:38307 every 8 connection(s)
I20260926 12:37:28.128800 29088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:37:28.129808 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 29088
I20260926 12:37:28.129941 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:37:28.130262 10632 external_mini_cluster.cc:1428] Setting key 2f3fc572ecb4045c25760dd417b7b7e9
Sep 26 12:37:28 dist-test-slave-t7tk krb5kdc[28713](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426248, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:37:28.141860 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:28.130972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:33569 (local address 127.10.98.62:41133)
0926 12:37:28.131131 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:28.131135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:28.132074 (+ 939us) server_negotiation.cc:407] Connection header received
0926 12:37:28.133058 (+ 984us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:28.133061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:28.133146 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:28.133220 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:28.134430 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.134961 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.136073 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.136307 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.136392 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:28.138662 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:28.138665 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:28.138668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:28.138694 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:28.139870 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:28.140346 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:28.140350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:28.140352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:28.140446 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:28.140743 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:28.140746 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:28.140748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:28.140943 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:28.141026 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:28.141560 (+ 534us) server_negotiation.cc:299] Negotiation successful
0926 12:37:28.141678 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 12:37:28.142689 29219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:28.131269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:41133 (local address 127.10.98.3:33569)
0926 12:37:28.131931 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:28.131971 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:28.132859 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:28.133418 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:28.133425 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:28.133783 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:28.134294 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:28.134309 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.135082 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.135085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:28.135953 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:28.135962 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.136444 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.136448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:28.136611 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:28.137138 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:37:28.137154 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:28.138508 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:28.139975 (+ 1467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:28.139981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:28.139998 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:28.140250 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:28.140525 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:28.140528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:28.140531 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:28.140659 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:28.141047 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:28.141054 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:28.141331 (+ 277us) client_negotiation.cc:769] Sending connection context
0926 12:37:28.141496 (+ 165us) client_negotiation.cc:240] Negotiation successful
0926 12:37:28.141815 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":113,"threads_started":1}
I20260926 12:37:28.143812 29217 heartbeater.cc:347] Connected to a master server at 127.10.98.62:41133
I20260926 12:37:28.144068 29217 heartbeater.cc:464] Registering TS with master...
I20260926 12:37:28.144573 29217 heartbeater.cc:511] Master 127.10.98.62:41133 requested a full tablet report, sending...
I20260926 12:37:28.145586 28757 ts_manager.cc:194] Registered new tserver with Master: a9a779dd515c4adba750eda2785058ff (127.10.98.3:38307)
I20260926 12:37:28.146134 28757 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:33569
I20260926 12:37:28.156037 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:37:28.165278 28797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:28.157967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35474 (local address 127.10.98.62:41133)
0926 12:37:28.158093 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:28.158097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:28.158241 (+ 144us) server_negotiation.cc:407] Connection header received
0926 12:37:28.158338 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:28.158341 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:28.158381 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:28.158444 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:28.159107 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.159628 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.160404 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.160662 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.160740 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:28.162203 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:28.162207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:28.162209 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:28.162234 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:28.163626 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:28.164115 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:28.164118 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:28.164119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:28.164196 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:28.164456 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:28.164458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:28.164459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:28.164665 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:28.164777 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:28.164978 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 12:37:28.165054 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:37:28.168275 28757 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:37:28.171424 28757 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:37:28.185222 29232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:28.179479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:38307 (local address 127.0.0.1:41778)
0926 12:37:28.180380 (+ 901us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:28.180394 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:28.180480 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:28.180995 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:28.181001 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:28.181037 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:28.181213 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:28.181220 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.182542 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.182545 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:28.183573 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:28.183583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.184651 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.184654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:28.184762 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:28.184800 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 12:37:28.184909 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 12:37:28.185025 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"mutex_wait_us":1,"thread_start_us":71,"threads_started":1}
I20260926 12:37:28.185222 29231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:28.180093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41778 (local address 127.10.98.3:38307)
0926 12:37:28.180501 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:28.180514 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:28.180550 (+ 36us) server_negotiation.cc:407] Connection header received
0926 12:37:28.180626 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:28.180630 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:28.180832 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:28.181359 (+ 527us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:28.181500 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.182415 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.183712 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.184547 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.184644 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:28.184791 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:28.184950 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 12:37:28.185044 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":108,"threads_started":1}
I20260926 12:37:28.186012 29229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:28.179414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:36357 (local address 127.0.0.1:59052)
0926 12:37:28.180251 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:28.180305 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:28.180438 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:28.181154 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:28.181157 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:28.181171 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:28.181889 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:28.181896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.183246 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.183249 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:28.184345 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:28.184354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.185551 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.185554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:28.185685 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:28.185709 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 12:37:28.185779 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 12:37:28.185848 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"mutex_wait_us":671,"thread_start_us":100,"threads_started":1}
I20260926 12:37:28.186055 29230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:28.179565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59052 (local address 127.10.98.1:36357)
0926 12:37:28.179824 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:28.179841 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:28.180590 (+ 749us) server_negotiation.cc:407] Connection header received
0926 12:37:28.180659 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:28.180663 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:28.180832 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:28.180974 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:28.182009 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.183115 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.184454 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.185250 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.185349 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:28.185482 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:28.185825 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 12:37:28.185916 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":86,"threads_started":1}
I20260926 12:37:28.186369 29233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:28.180475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.10.98.2:37905)
0926 12:37:28.180804 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:28.180815 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:28.181231 (+ 416us) server_negotiation.cc:407] Connection header received
0926 12:37:28.181315 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:28.181320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:28.181504 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:28.181651 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:28.181860 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.183077 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.184582 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.185391 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.185488 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:28.185588 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:28.186155 (+ 567us) server_negotiation.cc:299] Negotiation successful
0926 12:37:28.186245 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":143,"threads_started":1}
I20260926 12:37:28.186066 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:28.180166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:37905 (local address 127.0.0.1:47820)
0926 12:37:28.180988 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:28.181001 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:28.181132 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:28.181654 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:28.181656 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:28.181671 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:28.181747 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:28.181752 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.183219 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.183223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:28.184437 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:28.184448 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.185640 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.185643 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:28.185752 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:28.185773 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:37:28.185832 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 12:37:28.185902 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":765,"mutex_wait_us":53,"thread_start_us":97,"threads_started":1}
I20260926 12:37:28.189049 29017 tablet_service.cc:1467] Processing CreateTablet for tablet 81b4d471c2404e3ca9aa27a1bd40afaf (DEFAULT_TABLE table=test-table [id=8c6be140d23b4cd69801fe7df5dbb9c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:37:28.189213 29151 tablet_service.cc:1467] Processing CreateTablet for tablet 81b4d471c2404e3ca9aa27a1bd40afaf (DEFAULT_TABLE table=test-table [id=8c6be140d23b4cd69801fe7df5dbb9c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:37:28.189213 28883 tablet_service.cc:1467] Processing CreateTablet for tablet 81b4d471c2404e3ca9aa27a1bd40afaf (DEFAULT_TABLE table=test-table [id=8c6be140d23b4cd69801fe7df5dbb9c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:37:28.190050 29017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b4d471c2404e3ca9aa27a1bd40afaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:28.190204 28883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b4d471c2404e3ca9aa27a1bd40afaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:28.190210 29151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 81b4d471c2404e3ca9aa27a1bd40afaf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:28.196719 29234 tablet_bootstrap.cc:492] T 81b4d471c2404e3ca9aa27a1bd40afaf P c0b5972966ec4eefb08d92f0df7fdb37: Bootstrap starting.
I20260926 12:37:28.197412 29235 tablet_bootstrap.cc:492] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff: Bootstrap starting.
I20260926 12:37:28.197742 29236 tablet_bootstrap.cc:492] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a: Bootstrap starting.
I20260926 12:37:28.199061 29234 tablet_bootstrap.cc:654] T 81b4d471c2404e3ca9aa27a1bd40afaf P c0b5972966ec4eefb08d92f0df7fdb37: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:28.199980 29236 tablet_bootstrap.cc:654] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:28.200320 29234 log.cc:824] T 81b4d471c2404e3ca9aa27a1bd40afaf P c0b5972966ec4eefb08d92f0df7fdb37: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:28.201175 29236 log.cc:824] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:28.202306 29234 tablet_bootstrap.cc:492] T 81b4d471c2404e3ca9aa27a1bd40afaf P c0b5972966ec4eefb08d92f0df7fdb37: No bootstrap required, opened a new log
I20260926 12:37:28.202487 29234 ts_tablet_manager.cc:1397] T 81b4d471c2404e3ca9aa27a1bd40afaf P c0b5972966ec4eefb08d92f0df7fdb37: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:37:28.203081 29236 tablet_bootstrap.cc:492] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a: No bootstrap required, opened a new log
I20260926 12:37:28.203260 29236 ts_tablet_manager.cc:1397] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260926 12:37:28.203372 29235 tablet_bootstrap.cc:654] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:28.204504 29235 log.cc:824] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:28.205644 29234 raft_consensus.cc:348] T 81b4d471c2404e3ca9aa27a1bd40afaf P c0b5972966ec4eefb08d92f0df7fdb37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c98562bcaef4057a62fe6836f04710a" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36357 } } peers { permanent_uuid: "a9a779dd515c4adba750eda2785058ff" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38307 } } peers { permanent_uuid: "c0b5972966ec4eefb08d92f0df7fdb37" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37905 } }
I20260926 12:37:28.205910 29234 raft_consensus.cc:374] T 81b4d471c2404e3ca9aa27a1bd40afaf P c0b5972966ec4eefb08d92f0df7fdb37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:28.205992 29234 raft_consensus.cc:729] T 81b4d471c2404e3ca9aa27a1bd40afaf P c0b5972966ec4eefb08d92f0df7fdb37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0b5972966ec4eefb08d92f0df7fdb37, State: Initialized, Role: FOLLOWER
I20260926 12:37:28.206415 29234 consensus_queue.cc:260] T 81b4d471c2404e3ca9aa27a1bd40afaf P c0b5972966ec4eefb08d92f0df7fdb37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c98562bcaef4057a62fe6836f04710a" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36357 } } peers { permanent_uuid: "a9a779dd515c4adba750eda2785058ff" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38307 } } peers { permanent_uuid: "c0b5972966ec4eefb08d92f0df7fdb37" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37905 } }
I20260926 12:37:28.206526 29235 tablet_bootstrap.cc:492] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff: No bootstrap required, opened a new log
I20260926 12:37:28.206748 29235 ts_tablet_manager.cc:1397] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260926 12:37:28.207000 29083 heartbeater.cc:503] Master 127.10.98.62:41133 was elected leader, sending a full tablet report...
I20260926 12:37:28.206897 29236 raft_consensus.cc:348] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c98562bcaef4057a62fe6836f04710a" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36357 } } peers { permanent_uuid: "a9a779dd515c4adba750eda2785058ff" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38307 } } peers { permanent_uuid: "c0b5972966ec4eefb08d92f0df7fdb37" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37905 } }
I20260926 12:37:28.207202 29236 raft_consensus.cc:374] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:28.207301 29236 raft_consensus.cc:729] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c98562bcaef4057a62fe6836f04710a, State: Initialized, Role: FOLLOWER
I20260926 12:37:28.207365 29234 ts_tablet_manager.cc:1428] T 81b4d471c2404e3ca9aa27a1bd40afaf P c0b5972966ec4eefb08d92f0df7fdb37: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260926 12:37:28.207791 29236 consensus_queue.cc:260] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c98562bcaef4057a62fe6836f04710a" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36357 } } peers { permanent_uuid: "a9a779dd515c4adba750eda2785058ff" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38307 } } peers { permanent_uuid: "c0b5972966ec4eefb08d92f0df7fdb37" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37905 } }
I20260926 12:37:28.209002 29236 ts_tablet_manager.cc:1428] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260926 12:37:28.209378 28949 heartbeater.cc:503] Master 127.10.98.62:41133 was elected leader, sending a full tablet report...
I20260926 12:37:28.210466 29235 raft_consensus.cc:348] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c98562bcaef4057a62fe6836f04710a" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36357 } } peers { permanent_uuid: "a9a779dd515c4adba750eda2785058ff" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38307 } } peers { permanent_uuid: "c0b5972966ec4eefb08d92f0df7fdb37" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37905 } }
I20260926 12:37:28.210738 29235 raft_consensus.cc:374] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:28.210793 29235 raft_consensus.cc:729] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9a779dd515c4adba750eda2785058ff, State: Initialized, Role: FOLLOWER
I20260926 12:37:28.211244 29235 consensus_queue.cc:260] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c98562bcaef4057a62fe6836f04710a" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36357 } } peers { permanent_uuid: "a9a779dd515c4adba750eda2785058ff" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38307 } } peers { permanent_uuid: "c0b5972966ec4eefb08d92f0df7fdb37" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37905 } }
I20260926 12:37:28.211865 29217 heartbeater.cc:503] Master 127.10.98.62:41133 was elected leader, sending a full tablet report...
I20260926 12:37:28.212217 29235 ts_tablet_manager.cc:1428] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
W20260926 12:37:28.214322 28950 tablet.cc:2369] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:37:28.379804 29218 tablet.cc:2369] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 12:37:28.420864 29084 tablet.cc:2369] T 81b4d471c2404e3ca9aa27a1bd40afaf P c0b5972966ec4eefb08d92f0df7fdb37: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:37:28.450482 29241 raft_consensus.cc:482] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:37:28.450732 29241 raft_consensus.cc:504] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c98562bcaef4057a62fe6836f04710a" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36357 } } peers { permanent_uuid: "a9a779dd515c4adba750eda2785058ff" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38307 } } peers { permanent_uuid: "c0b5972966ec4eefb08d92f0df7fdb37" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37905 } }
I20260926 12:37:28.451917 29241 leader_election.cc:288] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9a779dd515c4adba750eda2785058ff (127.10.98.3:38307), c0b5972966ec4eefb08d92f0df7fdb37 (127.10.98.2:37905)
I20260926 12:37:28.456604 29231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:28.452256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:50673 (local address 127.10.98.3:38307)
0926 12:37:28.452427 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:28.452432 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:28.452531 (+ 99us) server_negotiation.cc:407] Connection header received
0926 12:37:28.452692 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:28.452695 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:28.452747 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:28.452821 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:28.453091 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.453854 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.455391 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.456027 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.456137 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:28.456205 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:28.456410 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 12:37:28.456476 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260926 12:37:28.456617 29233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:28.452635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:36529 (local address 127.10.98.2:37905)
0926 12:37:28.452774 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:28.452778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:28.452927 (+ 149us) server_negotiation.cc:407] Connection header received
0926 12:37:28.453013 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:28.453015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:28.453060 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:28.453171 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:28.453388 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.454089 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.455389 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.456036 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.456137 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:28.456205 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:28.456401 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 12:37:28.456476 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:37:28.456620 29244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:28.452513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:37905 (local address 127.10.98.1:36529)
0926 12:37:28.452837 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:28.452850 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:28.452939 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:28.453184 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:28.453187 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:28.453203 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:28.453293 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:28.453297 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.454189 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.454192 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:28.455285 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:28.455292 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.456144 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.456147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:28.456258 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:28.456285 (+ 27us) client_negotiation.cc:769] Sending connection context
0926 12:37:28.456358 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 12:37:28.456443 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"mutex_wait_us":3,"thread_start_us":102,"threads_started":1}
I20260926 12:37:28.456622 29243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:28.452142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:38307 (local address 127.10.98.1:50673)
0926 12:37:28.452426 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:28.452442 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:28.452585 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:28.452840 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:28.452843 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:28.452868 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:28.452989 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:28.452995 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.453969 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.453972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:28.455266 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:28.455277 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.456211 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.456214 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:28.456306 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:28.456323 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 12:37:28.456384 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 12:37:28.456443 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":1,"thread_start_us":90,"threads_started":1}
I20260926 12:37:28.457489 29037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b4d471c2404e3ca9aa27a1bd40afaf" candidate_uuid: "7c98562bcaef4057a62fe6836f04710a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0b5972966ec4eefb08d92f0df7fdb37" is_pre_election: true
I20260926 12:37:28.457494 29171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b4d471c2404e3ca9aa27a1bd40afaf" candidate_uuid: "7c98562bcaef4057a62fe6836f04710a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9a779dd515c4adba750eda2785058ff" is_pre_election: true
I20260926 12:37:28.457765 29037 raft_consensus.cc:2454] T 81b4d471c2404e3ca9aa27a1bd40afaf P c0b5972966ec4eefb08d92f0df7fdb37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c98562bcaef4057a62fe6836f04710a in term 0.
I20260926 12:37:28.457772 29171 raft_consensus.cc:2454] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c98562bcaef4057a62fe6836f04710a in term 0.
I20260926 12:37:28.458204 28836 leader_election.cc:302] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c98562bcaef4057a62fe6836f04710a, a9a779dd515c4adba750eda2785058ff; no voters:
I20260926 12:37:28.458461 29241 raft_consensus.cc:2781] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:37:28.458567 29241 raft_consensus.cc:482] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:37:28.458600 29241 raft_consensus.cc:3037] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:28.459591 29241 raft_consensus.cc:504] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c98562bcaef4057a62fe6836f04710a" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36357 } } peers { permanent_uuid: "a9a779dd515c4adba750eda2785058ff" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38307 } } peers { permanent_uuid: "c0b5972966ec4eefb08d92f0df7fdb37" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37905 } }
I20260926 12:37:28.459908 29241 leader_election.cc:288] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a [CANDIDATE]: Term 1 election: Requested vote from peers a9a779dd515c4adba750eda2785058ff (127.10.98.3:38307), c0b5972966ec4eefb08d92f0df7fdb37 (127.10.98.2:37905)
I20260926 12:37:28.460268 29171 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b4d471c2404e3ca9aa27a1bd40afaf" candidate_uuid: "7c98562bcaef4057a62fe6836f04710a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9a779dd515c4adba750eda2785058ff"
I20260926 12:37:28.460336 29037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "81b4d471c2404e3ca9aa27a1bd40afaf" candidate_uuid: "7c98562bcaef4057a62fe6836f04710a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0b5972966ec4eefb08d92f0df7fdb37"
I20260926 12:37:28.460387 29171 raft_consensus.cc:3037] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:28.460455 29037 raft_consensus.cc:3037] T 81b4d471c2404e3ca9aa27a1bd40afaf P c0b5972966ec4eefb08d92f0df7fdb37 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:28.461500 29037 raft_consensus.cc:2454] T 81b4d471c2404e3ca9aa27a1bd40afaf P c0b5972966ec4eefb08d92f0df7fdb37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c98562bcaef4057a62fe6836f04710a in term 1.
I20260926 12:37:28.461504 29171 raft_consensus.cc:2454] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c98562bcaef4057a62fe6836f04710a in term 1.
I20260926 12:37:28.461802 28836 leader_election.cc:302] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c98562bcaef4057a62fe6836f04710a, a9a779dd515c4adba750eda2785058ff; no voters:
I20260926 12:37:28.461963 29241 raft_consensus.cc:2781] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:28.462193 29241 raft_consensus.cc:686] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a [term 1 LEADER]: Becoming Leader. State: Replica: 7c98562bcaef4057a62fe6836f04710a, State: Running, Role: LEADER
I20260926 12:37:28.462541 29241 consensus_queue.cc:237] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c98562bcaef4057a62fe6836f04710a" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36357 } } peers { permanent_uuid: "a9a779dd515c4adba750eda2785058ff" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38307 } } peers { permanent_uuid: "c0b5972966ec4eefb08d92f0df7fdb37" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37905 } }
I20260926 12:37:28.465776 28758 catalog_manager.cc:5495] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c98562bcaef4057a62fe6836f04710a (127.10.98.1). New cstate: current_term: 1 leader_uuid: "7c98562bcaef4057a62fe6836f04710a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c98562bcaef4057a62fe6836f04710a" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 36357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a9a779dd515c4adba750eda2785058ff" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38307 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0b5972966ec4eefb08d92f0df7fdb37" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37905 } health_report { overall_health: UNKNOWN } } }
I20260926 12:37:28.477567 29230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:28.473925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59060 (local address 127.10.98.1:36357)
0926 12:37:28.474052 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:28.474056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:28.474087 (+ 31us) server_negotiation.cc:407] Connection header received
0926 12:37:28.474150 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:28.474152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:28.474206 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:28.474284 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:37:28.474588 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.475113 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.475974 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.476203 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.476297 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:28.476736 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:37:28.477208 (+ 472us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:37:28.477332 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:28.477417 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 12:37:28.477454 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 12:37:28.483415 29171 raft_consensus.cc:1261] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff [term 1 FOLLOWER]: Refusing update from remote peer 7c98562bcaef4057a62fe6836f04710a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:37:28.483408 29037 raft_consensus.cc:1261] T 81b4d471c2404e3ca9aa27a1bd40afaf P c0b5972966ec4eefb08d92f0df7fdb37 [term 1 FOLLOWER]: Refusing update from remote peer 7c98562bcaef4057a62fe6836f04710a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:37:28.484069 29241 consensus_queue.cc:1035] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9a779dd515c4adba750eda2785058ff" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 38307 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:37:28.484265 29245 consensus_queue.cc:1035] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0b5972966ec4eefb08d92f0df7fdb37" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:37:28.493255 29250 mvcc.cc:205] Tried to move back new op lower bound from 7333585913781305344 to 7333585913705152512. Current Snapshot: MvccSnapshot[applied={T|T < 7333585913781305344}]
I20260926 12:37:28.500005 28758 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:37:28.500202 28758 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:37:28.502646 28758 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a22e0a364b3d4f2a81373cb6996a651c: Sending DeleteTablet for 3 replicas of tablet 81b4d471c2404e3ca9aa27a1bd40afaf
I20260926 12:37:28.503396 28883 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b4d471c2404e3ca9aa27a1bd40afaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:37:28 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:59052
I20260926 12:37:28.503475 29017 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b4d471c2404e3ca9aa27a1bd40afaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:37:28 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:47820
I20260926 12:37:28.503558 29151 tablet_service.cc:1514] Processing DeleteTablet for tablet 81b4d471c2404e3ca9aa27a1bd40afaf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:37:28 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:41778
I20260926 12:37:28.503901 29259 tablet_replica.cc:330] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a: stopping tablet replica
I20260926 12:37:28.504221 29259 raft_consensus.cc:2229] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:37:28.504284 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 28820
I20260926 12:37:28.504604 29259 raft_consensus.cc:2258] T 81b4d471c2404e3ca9aa27a1bd40afaf P 7c98562bcaef4057a62fe6836f04710a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:37:28.506301 29261 tablet_replica.cc:330] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff: stopping tablet replica
I20260926 12:37:28.506596 29261 raft_consensus.cc:2229] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:37:28.506856 29260 tablet_replica.cc:330] T 81b4d471c2404e3ca9aa27a1bd40afaf P c0b5972966ec4eefb08d92f0df7fdb37: stopping tablet replica
I20260926 12:37:28.506920 29261 raft_consensus.cc:2258] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:37:28.507167 29260 raft_consensus.cc:2229] T 81b4d471c2404e3ca9aa27a1bd40afaf P c0b5972966ec4eefb08d92f0df7fdb37 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:37:28.507874 29260 raft_consensus.cc:2258] T 81b4d471c2404e3ca9aa27a1bd40afaf P c0b5972966ec4eefb08d92f0df7fdb37 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:37:28.508445 29261 ts_tablet_manager.cc:1905] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:37:28.509320 29260 ts_tablet_manager.cc:1905] T 81b4d471c2404e3ca9aa27a1bd40afaf P c0b5972966ec4eefb08d92f0df7fdb37: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 12:37:28.510181 28743 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.98.1:36357 (error 108)
I20260926 12:37:28.510958 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 28954
I20260926 12:37:28.511193 29261 ts_tablet_manager.cc:1918] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:37:28.511284 29261 log.cc:1196] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/81b4d471c2404e3ca9aa27a1bd40afaf
I20260926 12:37:28.511442 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:28.511102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:36357 (local address 127.0.0.1:59074)
0926 12:37:28.511293 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:28.511354 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.1:36357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260926 12:37:28.511572 29261 ts_tablet_manager.cc:1939] T 81b4d471c2404e3ca9aa27a1bd40afaf P a9a779dd515c4adba750eda2785058ff: Deleting consensus metadata
W20260926 12:37:28.512111 28743 catalog_manager.cc:4584] TS 7c98562bcaef4057a62fe6836f04710a (127.10.98.1:36357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 81b4d471c2404e3ca9aa27a1bd40afaf: Network error: Client connection negotiation failed: client connection to 127.10.98.1:36357: connect: Connection refused (error 111)
I20260926 12:37:28.512809 28742 catalog_manager.cc:4841] TS a9a779dd515c4adba750eda2785058ff (127.10.98.3:38307): tablet 81b4d471c2404e3ca9aa27a1bd40afaf (table test-table [id=8c6be140d23b4cd69801fe7df5dbb9c7]) successfully deleted
I20260926 12:37:28.516476 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 29088
I20260926 12:37:28.518000 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:28.516608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:37905 (local address 127.0.0.1:47834)
0926 12:37:28.517853 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:28.517911 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.2:37905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1156}
I20260926 12:37:28.521230 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 28726
2026-09-26T12:37:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2378 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T12:37:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:28Z Disabled control of system clock
I20260926 12:37:28.549599 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:44393
--webserver_interface=127.10.98.62
--webserver_port=0
--builtin_ntp_servers=127.10.98.20:45637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:44393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 12:37:28.660979 29266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:28.661332 29266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:28.661375 29266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:28.665472 29266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:28.665539 29266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:28.665556 29266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:28.665575 29266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:28.665593 29266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:28.670493 29266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:45637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:44393
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:44393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:28.671542 29266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:28.672905 29266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:28.681449 29272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:28.681483 29271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:28.681649 29266 server_base.cc:1034] running on GCE node
W20260926 12:37:28.681552 29274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:28.682377 29266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:28.683441 29266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:28.684613 29266 hybrid_clock.cc:648] HybridClock initialized: now 1790426248684586 us; error 38 us; skew 500 ppm
I20260926 12:37:28.686578 29266 webserver.cc:460] Webserver started at http://127.10.98.62:34251/ using document root <none> and password file <none>
I20260926 12:37:28.687120 29266 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:28.687184 29266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:28.687363 29266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:28.689468 29266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "b5d8685d978a41159a8aa4f260c2bff4"
format_stamp: "Formatted at 2026-09-26 12:37:28 on dist-test-slave-t7tk"
server_key: "08f00ca26abbe673c17be8eb93e21ff8"
server_key_iv: "a426f8b4e07721ace52d63381ae3f30b"
server_key_version: "encryptionkey@0"
I20260926 12:37:28.690006 29266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "b5d8685d978a41159a8aa4f260c2bff4"
format_stamp: "Formatted at 2026-09-26 12:37:28 on dist-test-slave-t7tk"
server_key: "08f00ca26abbe673c17be8eb93e21ff8"
server_key_iv: "a426f8b4e07721ace52d63381ae3f30b"
server_key_version: "encryptionkey@0"
I20260926 12:37:28.693804 29266 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 12:37:28.696345 29280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:28.697700 29266 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:37:28.697819 29266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "b5d8685d978a41159a8aa4f260c2bff4"
format_stamp: "Formatted at 2026-09-26 12:37:28 on dist-test-slave-t7tk"
server_key: "08f00ca26abbe673c17be8eb93e21ff8"
server_key_iv: "a426f8b4e07721ace52d63381ae3f30b"
server_key_version: "encryptionkey@0"
I20260926 12:37:28.697922 29266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:28.720664 29266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:28.721410 29266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:28.721596 29266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:28.728451 29266 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:44393
I20260926 12:37:28.728497 29332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:44393 every 8 connection(s)
I20260926 12:37:28.729431 29266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:28.732513 29333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:28.735673 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 29266
I20260926 12:37:28.735772 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:28.736052 10632 external_mini_cluster.cc:1428] Setting key 22da26884091cc59eb51c2c1b9c835d2
I20260926 12:37:28.738477 29333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4: Bootstrap starting.
I20260926 12:37:28.740638 29333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:28.741401 29333 log.cc:824] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:28.743219 29333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4: No bootstrap required, opened a new log
I20260926 12:37:28.745275 29336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:28.737561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55844 (local address 127.10.98.62:44393)
0926 12:37:28.738202 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:28.738217 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:28.738252 (+ 35us) server_negotiation.cc:407] Connection header received
0926 12:37:28.738911 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:28.738928 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:28.739328 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:28.739636 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:28.740065 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.741124 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.741991 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.742517 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.742669 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:28.743573 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:28.743578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:28.743591 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:28.743973 (+ 382us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:28.744031 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:28.744059 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:28.744141 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:28.744280 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 12:37:28.744528 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":122,"threads_started":1}
I20260926 12:37:28.746706 29333 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d8685d978a41159a8aa4f260c2bff4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 44393 } }
I20260926 12:37:28.746922 29333 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:28.746959 29333 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5d8685d978a41159a8aa4f260c2bff4, State: Initialized, Role: FOLLOWER
I20260926 12:37:28.747364 29333 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d8685d978a41159a8aa4f260c2bff4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 44393 } }
I20260926 12:37:28.747506 29333 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:28.747566 29333 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:28.747658 29333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:28.748512 29333 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d8685d978a41159a8aa4f260c2bff4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 44393 } }
I20260926 12:37:28.748852 29333 leader_election.cc:302] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5d8685d978a41159a8aa4f260c2bff4; no voters:
I20260926 12:37:28.749148 29333 leader_election.cc:288] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:28.749248 29338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:28.749468 29338 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4 [term 1 LEADER]: Becoming Leader. State: Replica: b5d8685d978a41159a8aa4f260c2bff4, State: Running, Role: LEADER
I20260926 12:37:28.749820 29338 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d8685d978a41159a8aa4f260c2bff4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 44393 } }
I20260926 12:37:28.750289 29333 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:28.751353 29340 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5d8685d978a41159a8aa4f260c2bff4. Latest consensus state: current_term: 1 leader_uuid: "b5d8685d978a41159a8aa4f260c2bff4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d8685d978a41159a8aa4f260c2bff4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 44393 } } }
I20260926 12:37:28.751478 29340 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:28.751719 29339 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5d8685d978a41159a8aa4f260c2bff4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5d8685d978a41159a8aa4f260c2bff4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 44393 } } }
I20260926 12:37:28.751808 29339 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:28.751906 29347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:28.755007 29347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:28.760874 29347 catalog_manager.cc:1347] Generated new cluster ID: 4736a5059cb642c099ca753347e2bea0
I20260926 12:37:28.760942 29347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:28.775215 29347 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:28.776489 29347 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:28.787456 29347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4: Generated new TSK 0
I20260926 12:37:28.788316 29347 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:37:28.797030 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:44393
--builtin_ntp_servers=127.10.98.20:45637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 12:37:28.905946 29357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:28.906206 29357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:28.906242 29357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:28.910120 29357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:28.910169 29357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:28.910219 29357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:37:28.914865 29357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:45637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:44393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:28.915827 29357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:28.917091 29357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:28.926659 29362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:28.926756 29357 server_base.cc:1034] running on GCE node
W20260926 12:37:28.926609 29365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:28.926609 29363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:28.927340 29357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:28.927975 29357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:28.929159 29357 hybrid_clock.cc:648] HybridClock initialized: now 1790426248929128 us; error 40 us; skew 500 ppm
I20260926 12:37:28.930945 29357 webserver.cc:460] Webserver started at http://127.10.98.1:43173/ using document root <none> and password file <none>
I20260926 12:37:28.931488 29357 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:28.931561 29357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:28.931754 29357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:28.933830 29357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "1edcb57dc787409fbb730c92aaf0e86e"
format_stamp: "Formatted at 2026-09-26 12:37:28 on dist-test-slave-t7tk"
server_key: "740e9030cbcede502be8fcbac4479b6c"
server_key_iv: "439f4b9c9072666dbc5186d62edb18f1"
server_key_version: "encryptionkey@0"
I20260926 12:37:28.934352 29357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "1edcb57dc787409fbb730c92aaf0e86e"
format_stamp: "Formatted at 2026-09-26 12:37:28 on dist-test-slave-t7tk"
server_key: "740e9030cbcede502be8fcbac4479b6c"
server_key_iv: "439f4b9c9072666dbc5186d62edb18f1"
server_key_version: "encryptionkey@0"
I20260926 12:37:28.937939 29357 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:37:28.940267 29371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:28.941604 29357 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 12:37:28.941717 29357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "1edcb57dc787409fbb730c92aaf0e86e"
format_stamp: "Formatted at 2026-09-26 12:37:28 on dist-test-slave-t7tk"
server_key: "740e9030cbcede502be8fcbac4479b6c"
server_key_iv: "439f4b9c9072666dbc5186d62edb18f1"
server_key_version: "encryptionkey@0"
I20260926 12:37:28.941809 29357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:28.954284 29357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:28.954957 29357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:28.955124 29357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:28.955695 29357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:37:28.956610 29357 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:37:28.956655 29357 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:28.956709 29357 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:37:28.956736 29357 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:28.965221 29357 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:34833
I20260926 12:37:28.965286 29484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:34833 every 8 connection(s)
I20260926 12:37:28.966148 29357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:37:28.973222 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 29357
I20260926 12:37:28.973362 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:37:28.973652 10632 external_mini_cluster.cc:1428] Setting key 5e24ba1ae1e4f47a01c2d690ee6db146
I20260926 12:37:28.975706 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:44393
--builtin_ntp_servers=127.10.98.20:45637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 12:37:28.976107 29336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:28.967944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:41981 (local address 127.10.98.62:44393)
0926 12:37:28.968060 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:28.968063 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:28.969266 (+ 1203us) server_negotiation.cc:407] Connection header received
0926 12:37:28.970739 (+ 1473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:28.970742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:28.970791 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:28.970886 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:28.971882 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.972501 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.974082 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.974384 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.974494 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:28.975664 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:28.975667 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:28.975669 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:28.975681 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:28.975701 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:28.975708 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:28.975769 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:28.975943 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 12:37:28.975997 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 12:37:28.977238 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:28.968365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:44393 (local address 127.10.98.1:41981)
0926 12:37:28.969074 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:28.969152 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:28.970511 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:28.971037 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:28.971043 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:28.971129 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:28.971700 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:28.971715 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.972652 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.972654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:28.973931 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:28.973941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:28.974571 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:28.974574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:28.974724 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:28.975342 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:37:28.975343 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:28.975397 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:37:28.975479 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:28.975791 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:28.975801 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:28.975807 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 12:37:28.975934 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 12:37:28.976229 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":144,"threads_started":1}
I20260926 12:37:28.978406 29485 heartbeater.cc:347] Connected to a master server at 127.10.98.62:44393
I20260926 12:37:28.978641 29485 heartbeater.cc:464] Registering TS with master...
I20260926 12:37:28.979211 29485 heartbeater.cc:511] Master 127.10.98.62:44393 requested a full tablet report, sending...
I20260926 12:37:28.980847 29297 ts_manager.cc:194] Registered new tserver with Master: 1edcb57dc787409fbb730c92aaf0e86e (127.10.98.1:34833)
I20260926 12:37:28.982442 29297 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.98.1:41981
W20260926 12:37:29.086897 29488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:29.087184 29488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:29.087224 29488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:29.091246 29488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:29.091310 29488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:29.091379 29488 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:37:29.096204 29488 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:45637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:44393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29488
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:29.097317 29488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:29.098664 29488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:29.108055 29496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:29.107995 29493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:29.107993 29494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:29.108306 29488 server_base.cc:1034] running on GCE node
I20260926 12:37:29.108832 29488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:29.109490 29488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:29.110654 29488 hybrid_clock.cc:648] HybridClock initialized: now 1790426249110624 us; error 42 us; skew 500 ppm
I20260926 12:37:29.112484 29488 webserver.cc:460] Webserver started at http://127.10.98.2:43663/ using document root <none> and password file <none>
I20260926 12:37:29.113023 29488 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:29.113080 29488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:29.113289 29488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:29.115255 29488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "1b52fec9cc354413af981dd513fab89f"
format_stamp: "Formatted at 2026-09-26 12:37:29 on dist-test-slave-t7tk"
server_key: "3550a7218d3d7c518e53e3e9485b8217"
server_key_iv: "14c9906391a85c71730acded1d9bf41c"
server_key_version: "encryptionkey@0"
I20260926 12:37:29.115779 29488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "1b52fec9cc354413af981dd513fab89f"
format_stamp: "Formatted at 2026-09-26 12:37:29 on dist-test-slave-t7tk"
server_key: "3550a7218d3d7c518e53e3e9485b8217"
server_key_iv: "14c9906391a85c71730acded1d9bf41c"
server_key_version: "encryptionkey@0"
I20260926 12:37:29.119273 29488 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:37:29.121729 29502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:29.123193 29488 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:37:29.123373 29488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "1b52fec9cc354413af981dd513fab89f"
format_stamp: "Formatted at 2026-09-26 12:37:29 on dist-test-slave-t7tk"
server_key: "3550a7218d3d7c518e53e3e9485b8217"
server_key_iv: "14c9906391a85c71730acded1d9bf41c"
server_key_version: "encryptionkey@0"
I20260926 12:37:29.123481 29488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:29.144619 29488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:29.145452 29488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:29.145666 29488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:29.146330 29488 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:37:29.147358 29488 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:37:29.147413 29488 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:29.147473 29488 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:37:29.147506 29488 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:29.157047 29488 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:43795
I20260926 12:37:29.157083 29615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:43795 every 8 connection(s)
I20260926 12:37:29.158108 29488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:37:29.162667 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 29488
I20260926 12:37:29.162791 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:37:29.163120 10632 external_mini_cluster.cc:1428] Setting key 1f7a8d0ba717567ba479c9c36271a83d
I20260926 12:37:29.165341 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:44393
--builtin_ntp_servers=127.10.98.20:45637
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 12:37:29.168190 29336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:29.159953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:55223 (local address 127.10.98.62:44393)
0926 12:37:29.160080 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:29.160084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:29.161196 (+ 1112us) server_negotiation.cc:407] Connection header received
0926 12:37:29.162612 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:29.162615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:29.162665 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:29.162743 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:29.163939 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.164659 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.166308 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.166542 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.166660 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:29.167738 (+ 1078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:29.167742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:29.167744 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:29.167758 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:29.167780 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:29.167790 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:29.167858 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:29.167979 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 12:37:29.168044 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:37:29.169078 29618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:29.160350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:44393 (local address 127.10.98.2:55223)
0926 12:37:29.161011 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:29.161067 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:29.162358 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:29.162967 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:29.162977 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:29.163117 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:29.163754 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:29.163773 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.164793 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.164796 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:29.166172 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:29.166183 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.166726 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.166730 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:29.166902 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:29.167471 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:37:29.167473 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:29.167517 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:37:29.167588 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:29.167863 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:29.167869 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:29.167873 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:37:29.167966 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 12:37:29.168213 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":111,"threads_started":1}
I20260926 12:37:29.170264 29616 heartbeater.cc:347] Connected to a master server at 127.10.98.62:44393
I20260926 12:37:29.170491 29616 heartbeater.cc:464] Registering TS with master...
I20260926 12:37:29.171001 29616 heartbeater.cc:511] Master 127.10.98.62:44393 requested a full tablet report, sending...
I20260926 12:37:29.172024 29297 ts_manager.cc:194] Registered new tserver with Master: 1b52fec9cc354413af981dd513fab89f (127.10.98.2:43795)
I20260926 12:37:29.172597 29297 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.98.2:55223
W20260926 12:37:29.273712 29619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:29.273973 29619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:29.274012 29619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:29.278033 29619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:29.278088 29619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:29.278148 29619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:37:29.282820 29619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:45637
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:44393
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:29.283843 29619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:29.285167 29619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:29.294385 29624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:29.294358 29625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:29.294512 29619 server_base.cc:1034] running on GCE node
W20260926 12:37:29.294385 29627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:29.295068 29619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:29.295653 29619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:29.296804 29619 hybrid_clock.cc:648] HybridClock initialized: now 1790426249296786 us; error 34 us; skew 500 ppm
I20260926 12:37:29.298619 29619 webserver.cc:460] Webserver started at http://127.10.98.3:40141/ using document root <none> and password file <none>
I20260926 12:37:29.299137 29619 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:29.299196 29619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:29.299386 29619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:29.301487 29619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "cebfe71f687840278e5d42cb21930c47"
format_stamp: "Formatted at 2026-09-26 12:37:29 on dist-test-slave-t7tk"
server_key: "057f089bed7cc820eeee2975915f11af"
server_key_iv: "d2f14d7de7a3e4b6da53c58ef817267b"
server_key_version: "encryptionkey@0"
I20260926 12:37:29.302045 29619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "cebfe71f687840278e5d42cb21930c47"
format_stamp: "Formatted at 2026-09-26 12:37:29 on dist-test-slave-t7tk"
server_key: "057f089bed7cc820eeee2975915f11af"
server_key_iv: "d2f14d7de7a3e4b6da53c58ef817267b"
server_key_version: "encryptionkey@0"
I20260926 12:37:29.305866 29619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:37:29.308265 29633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:29.309508 29619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:37:29.309651 29619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "cebfe71f687840278e5d42cb21930c47"
format_stamp: "Formatted at 2026-09-26 12:37:29 on dist-test-slave-t7tk"
server_key: "057f089bed7cc820eeee2975915f11af"
server_key_iv: "d2f14d7de7a3e4b6da53c58ef817267b"
server_key_version: "encryptionkey@0"
I20260926 12:37:29.309751 29619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:29.335490 29619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:29.336196 29619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:29.336364 29619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:29.336962 29619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:37:29.337957 29619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:37:29.338006 29619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:29.338061 29619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:37:29.338088 29619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:29.346357 29619 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:45825
I20260926 12:37:29.346385 29746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:45825 every 8 connection(s)
I20260926 12:37:29.347236 29619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:37:29.351562 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 29619
I20260926 12:37:29.351673 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:37:29.351919 10632 external_mini_cluster.cc:1428] Setting key 2f5522b1c756e20ac4c4035fbb753b85
I20260926 12:37:29.356643 29336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:29.349067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:44655 (local address 127.10.98.62:44393)
0926 12:37:29.349239 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:29.349244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:29.350340 (+ 1096us) server_negotiation.cc:407] Connection header received
0926 12:37:29.351783 (+ 1443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:29.351786 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:29.351845 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:29.351921 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:29.352982 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.353661 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.354796 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.355018 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.355107 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:29.356157 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:29.356161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:29.356164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:29.356176 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:29.356196 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:29.356204 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:29.356262 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:29.356407 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 12:37:29.356470 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260926 12:37:29.357512 29749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:29.349505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:44393 (local address 127.10.98.3:44655)
0926 12:37:29.350186 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:29.350230 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:29.351562 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:29.352115 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:29.352123 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:29.352222 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:29.352843 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:29.352863 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.353788 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.353792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:29.354674 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:29.354683 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.355160 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.355164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:29.355327 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:29.355887 (+ 560us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:37:29.355889 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:29.355930 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:37:29.355994 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:29.356271 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:29.356279 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:29.356285 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 12:37:29.356397 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 12:37:29.356678 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":109,"threads_started":1}
I20260926 12:37:29.358559 29747 heartbeater.cc:347] Connected to a master server at 127.10.98.62:44393
I20260926 12:37:29.358767 29747 heartbeater.cc:464] Registering TS with master...
I20260926 12:37:29.359257 29747 heartbeater.cc:511] Master 127.10.98.62:44393 requested a full tablet report, sending...
I20260926 12:37:29.360183 29297 ts_manager.cc:194] Registered new tserver with Master: cebfe71f687840278e5d42cb21930c47 (127.10.98.3:45825)
I20260926 12:37:29.360751 29297 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.98.3:44655
I20260926 12:37:29.365761 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:37:29.371399 29336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:29.367652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55848 (local address 127.10.98.62:44393)
0926 12:37:29.367836 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:29.367841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:29.368009 (+ 168us) server_negotiation.cc:407] Connection header received
0926 12:37:29.368086 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:29.368089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:29.368136 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:29.368201 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:29.368441 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.368937 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.369789 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.370009 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.370095 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:29.370929 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:29.370932 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:29.370934 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:29.370946 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:29.370965 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:29.370972 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:29.371073 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:29.371199 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 12:37:29.371238 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 12:37:29.374219 29297 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55848:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:37:29.377233 29297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:37:29.390931 29758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:29.385279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:34833 (local address 127.0.0.1:46136)
0926 12:37:29.385613 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:29.385663 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:29.385801 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:29.386430 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:29.386437 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:29.386460 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:29.386649 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:29.386658 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.388198 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.388202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:29.389312 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:29.389323 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.390366 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.390369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:29.390484 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:29.390538 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 12:37:29.390650 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 12:37:29.390775 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":20,"thread_start_us":84,"threads_started":1}
I20260926 12:37:29.390969 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:29.385449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.10.98.1:34833)
0926 12:37:29.385902 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:29.385915 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:29.385940 (+ 25us) server_negotiation.cc:407] Connection header received
0926 12:37:29.386020 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:29.386026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:29.386248 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:29.386813 (+ 565us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:29.386951 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.388067 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.389456 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.390240 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.390344 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:29.390531 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:29.390692 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 12:37:29.390792 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":133,"threads_started":1}
I20260926 12:37:29.391575 29760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:29.385967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:45825 (local address 127.0.0.1:36180)
0926 12:37:29.386530 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:29.386541 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:29.386614 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:29.387399 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:29.387402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:29.387419 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:29.387497 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:29.387503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.388599 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.388602 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:29.389896 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:29.389902 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.390806 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.390810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:29.390895 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:29.390918 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:37:29.390961 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 12:37:29.391449 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"mutex_wait_us":91,"thread_start_us":109,"threads_started":1}
I20260926 12:37:29.391620 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:29.386140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37658 (local address 127.10.98.2:43795)
0926 12:37:29.386551 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:29.386565 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:29.386587 (+ 22us) server_negotiation.cc:407] Connection header received
0926 12:37:29.386661 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:29.386666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:29.386817 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:29.386974 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:29.387338 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.388417 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.390031 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.390774 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.390872 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:29.391046 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:29.391362 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 12:37:29.391459 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":111,"threads_started":1}
I20260926 12:37:29.391897 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:29.385707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:43795 (local address 127.0.0.1:37658)
0926 12:37:29.386249 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:29.386269 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:29.386353 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:29.387107 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:29.387110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:29.387126 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:29.387218 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:29.387224 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.388532 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.388535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:29.389890 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:29.389901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.391116 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.391119 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:29.391210 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:29.391232 (+ 22us) client_negotiation.cc:769] Sending connection context
0926 12:37:29.391309 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 12:37:29.391785 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":112,"thread_start_us":104,"threads_started":1}
I20260926 12:37:29.392186 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:29.386687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36180 (local address 127.10.98.3:45825)
0926 12:37:29.387002 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:29.387013 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:29.387035 (+ 22us) server_negotiation.cc:407] Connection header received
0926 12:37:29.387106 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:29.387110 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:29.387277 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:29.387412 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:29.387583 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.388447 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.390041 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.390688 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.391470 (+ 782us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:29.391889 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:29.391941 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 12:37:29.392038 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":99,"threads_started":1}
I20260926 12:37:29.394275 29550 tablet_service.cc:1467] Processing CreateTablet for tablet 27450662feaf4d47ba09c9d9d45ef075 (DEFAULT_TABLE table=test-table [id=0dd47359e3e940a3ae68d7b1e2e312c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:37:29.394145 29419 tablet_service.cc:1467] Processing CreateTablet for tablet 27450662feaf4d47ba09c9d9d45ef075 (DEFAULT_TABLE table=test-table [id=0dd47359e3e940a3ae68d7b1e2e312c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:37:29.394660 29681 tablet_service.cc:1467] Processing CreateTablet for tablet 27450662feaf4d47ba09c9d9d45ef075 (DEFAULT_TABLE table=test-table [id=0dd47359e3e940a3ae68d7b1e2e312c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:37:29.395205 29550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27450662feaf4d47ba09c9d9d45ef075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:29.395212 29419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27450662feaf4d47ba09c9d9d45ef075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:29.395514 29681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27450662feaf4d47ba09c9d9d45ef075. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:29.401029 29764 tablet_bootstrap.cc:492] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f: Bootstrap starting.
I20260926 12:37:29.402484 29766 tablet_bootstrap.cc:492] T 27450662feaf4d47ba09c9d9d45ef075 P 1edcb57dc787409fbb730c92aaf0e86e: Bootstrap starting.
I20260926 12:37:29.402868 29765 tablet_bootstrap.cc:492] T 27450662feaf4d47ba09c9d9d45ef075 P cebfe71f687840278e5d42cb21930c47: Bootstrap starting.
I20260926 12:37:29.403896 29764 tablet_bootstrap.cc:654] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:29.404588 29766 tablet_bootstrap.cc:654] T 27450662feaf4d47ba09c9d9d45ef075 P 1edcb57dc787409fbb730c92aaf0e86e: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:29.404577 29764 log.cc:824] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:29.404680 29765 tablet_bootstrap.cc:654] T 27450662feaf4d47ba09c9d9d45ef075 P cebfe71f687840278e5d42cb21930c47: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:29.405432 29766 log.cc:824] T 27450662feaf4d47ba09c9d9d45ef075 P 1edcb57dc787409fbb730c92aaf0e86e: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:29.405431 29765 log.cc:824] T 27450662feaf4d47ba09c9d9d45ef075 P cebfe71f687840278e5d42cb21930c47: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:29.406491 29764 tablet_bootstrap.cc:492] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f: No bootstrap required, opened a new log
I20260926 12:37:29.406670 29764 ts_tablet_manager.cc:1397] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20260926 12:37:29.407169 29765 tablet_bootstrap.cc:492] T 27450662feaf4d47ba09c9d9d45ef075 P cebfe71f687840278e5d42cb21930c47: No bootstrap required, opened a new log
I20260926 12:37:29.407346 29765 ts_tablet_manager.cc:1397] T 27450662feaf4d47ba09c9d9d45ef075 P cebfe71f687840278e5d42cb21930c47: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260926 12:37:29.407606 29766 tablet_bootstrap.cc:492] T 27450662feaf4d47ba09c9d9d45ef075 P 1edcb57dc787409fbb730c92aaf0e86e: No bootstrap required, opened a new log
I20260926 12:37:29.407778 29766 ts_tablet_manager.cc:1397] T 27450662feaf4d47ba09c9d9d45ef075 P 1edcb57dc787409fbb730c92aaf0e86e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:37:29.410238 29764 raft_consensus.cc:348] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edcb57dc787409fbb730c92aaf0e86e" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34833 } } peers { permanent_uuid: "cebfe71f687840278e5d42cb21930c47" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 45825 } } peers { permanent_uuid: "1b52fec9cc354413af981dd513fab89f" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 43795 } }
I20260926 12:37:29.410455 29764 raft_consensus.cc:374] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:29.410507 29764 raft_consensus.cc:729] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b52fec9cc354413af981dd513fab89f, State: Initialized, Role: FOLLOWER
I20260926 12:37:29.410900 29764 consensus_queue.cc:260] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edcb57dc787409fbb730c92aaf0e86e" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34833 } } peers { permanent_uuid: "cebfe71f687840278e5d42cb21930c47" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 45825 } } peers { permanent_uuid: "1b52fec9cc354413af981dd513fab89f" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 43795 } }
I20260926 12:37:29.411453 29765 raft_consensus.cc:348] T 27450662feaf4d47ba09c9d9d45ef075 P cebfe71f687840278e5d42cb21930c47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edcb57dc787409fbb730c92aaf0e86e" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34833 } } peers { permanent_uuid: "cebfe71f687840278e5d42cb21930c47" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 45825 } } peers { permanent_uuid: "1b52fec9cc354413af981dd513fab89f" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 43795 } }
I20260926 12:37:29.411538 29766 raft_consensus.cc:348] T 27450662feaf4d47ba09c9d9d45ef075 P 1edcb57dc787409fbb730c92aaf0e86e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edcb57dc787409fbb730c92aaf0e86e" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34833 } } peers { permanent_uuid: "cebfe71f687840278e5d42cb21930c47" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 45825 } } peers { permanent_uuid: "1b52fec9cc354413af981dd513fab89f" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 43795 } }
I20260926 12:37:29.411705 29765 raft_consensus.cc:374] T 27450662feaf4d47ba09c9d9d45ef075 P cebfe71f687840278e5d42cb21930c47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:29.411720 29766 raft_consensus.cc:374] T 27450662feaf4d47ba09c9d9d45ef075 P 1edcb57dc787409fbb730c92aaf0e86e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:29.411715 29764 ts_tablet_manager.cc:1428] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f: Time spent starting tablet: real 0.005s user 0.004s sys 0.001s
I20260926 12:37:29.411767 29765 raft_consensus.cc:729] T 27450662feaf4d47ba09c9d9d45ef075 P cebfe71f687840278e5d42cb21930c47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cebfe71f687840278e5d42cb21930c47, State: Initialized, Role: FOLLOWER
I20260926 12:37:29.411767 29766 raft_consensus.cc:729] T 27450662feaf4d47ba09c9d9d45ef075 P 1edcb57dc787409fbb730c92aaf0e86e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1edcb57dc787409fbb730c92aaf0e86e, State: Initialized, Role: FOLLOWER
I20260926 12:37:29.412050 29616 heartbeater.cc:503] Master 127.10.98.62:44393 was elected leader, sending a full tablet report...
I20260926 12:37:29.412213 29765 consensus_queue.cc:260] T 27450662feaf4d47ba09c9d9d45ef075 P cebfe71f687840278e5d42cb21930c47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edcb57dc787409fbb730c92aaf0e86e" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34833 } } peers { permanent_uuid: "cebfe71f687840278e5d42cb21930c47" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 45825 } } peers { permanent_uuid: "1b52fec9cc354413af981dd513fab89f" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 43795 } }
I20260926 12:37:29.412213 29766 consensus_queue.cc:260] T 27450662feaf4d47ba09c9d9d45ef075 P 1edcb57dc787409fbb730c92aaf0e86e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edcb57dc787409fbb730c92aaf0e86e" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34833 } } peers { permanent_uuid: "cebfe71f687840278e5d42cb21930c47" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 45825 } } peers { permanent_uuid: "1b52fec9cc354413af981dd513fab89f" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 43795 } }
I20260926 12:37:29.413070 29747 heartbeater.cc:503] Master 127.10.98.62:44393 was elected leader, sending a full tablet report...
I20260926 12:37:29.413141 29766 ts_tablet_manager.cc:1428] T 27450662feaf4d47ba09c9d9d45ef075 P 1edcb57dc787409fbb730c92aaf0e86e: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260926 12:37:29.413177 29765 ts_tablet_manager.cc:1428] T 27450662feaf4d47ba09c9d9d45ef075 P cebfe71f687840278e5d42cb21930c47: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260926 12:37:29.414095 29485 heartbeater.cc:503] Master 127.10.98.62:44393 was elected leader, sending a full tablet report...
W20260926 12:37:29.467304 29486 tablet.cc:2369] T 27450662feaf4d47ba09c9d9d45ef075 P 1edcb57dc787409fbb730c92aaf0e86e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:37:29.492457 29770 raft_consensus.cc:482] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:37:29.492714 29770 raft_consensus.cc:504] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edcb57dc787409fbb730c92aaf0e86e" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34833 } } peers { permanent_uuid: "cebfe71f687840278e5d42cb21930c47" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 45825 } } peers { permanent_uuid: "1b52fec9cc354413af981dd513fab89f" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 43795 } }
I20260926 12:37:29.493852 29770 leader_election.cc:288] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1edcb57dc787409fbb730c92aaf0e86e (127.10.98.1:34833), cebfe71f687840278e5d42cb21930c47 (127.10.98.3:45825)
I20260926 12:37:29.497576 29759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:29.494218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:52485 (local address 127.10.98.1:34833)
0926 12:37:29.494348 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:29.494353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:29.494368 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:37:29.494651 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:29.494654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:29.494709 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:29.494786 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:29.495128 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.495628 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.496661 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.497035 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.497158 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:29.497222 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:29.497384 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 12:37:29.497454 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 12:37:29.497577 29618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:29.494113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:34833 (local address 127.10.98.2:52485)
0926 12:37:29.494249 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:29.494266 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:29.494372 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:29.494830 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:29.494833 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:29.494870 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:29.495007 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:29.495013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.495728 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.495732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:29.496527 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:29.496545 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.497158 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.497162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:29.497272 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:29.497293 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:37:29.497357 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 12:37:29.497435 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"spinlock_wait_cycles":2944}
I20260926 12:37:29.497857 29763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:29.494526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:59387 (local address 127.10.98.3:45825)
0926 12:37:29.494673 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:29.494677 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:29.494829 (+ 152us) server_negotiation.cc:407] Connection header received
0926 12:37:29.494978 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:29.494981 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:29.495021 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:29.495103 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:29.495361 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.495868 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.496871 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.497325 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.497420 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:29.497476 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:29.497656 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 12:37:29.497735 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:37:29.497959 29773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:29.494390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:45825 (local address 127.10.98.2:59387)
0926 12:37:29.494734 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:29.494746 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:29.494852 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:29.495121 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:29.495124 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:29.495139 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:29.495261 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:29.495267 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.495966 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.495970 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:29.496754 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:29.496762 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.497446 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.497450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:29.497551 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:29.497568 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 12:37:29.497762 (+ 194us) client_negotiation.cc:240] Negotiation successful
0926 12:37:29.497831 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":23,"thread_start_us":113,"threads_started":1}
I20260926 12:37:29.498368 29439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27450662feaf4d47ba09c9d9d45ef075" candidate_uuid: "1b52fec9cc354413af981dd513fab89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1edcb57dc787409fbb730c92aaf0e86e" is_pre_election: true
I20260926 12:37:29.498627 29701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27450662feaf4d47ba09c9d9d45ef075" candidate_uuid: "1b52fec9cc354413af981dd513fab89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cebfe71f687840278e5d42cb21930c47" is_pre_election: true
I20260926 12:37:29.498668 29439 raft_consensus.cc:2454] T 27450662feaf4d47ba09c9d9d45ef075 P 1edcb57dc787409fbb730c92aaf0e86e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b52fec9cc354413af981dd513fab89f in term 0.
I20260926 12:37:29.498922 29701 raft_consensus.cc:2454] T 27450662feaf4d47ba09c9d9d45ef075 P cebfe71f687840278e5d42cb21930c47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b52fec9cc354413af981dd513fab89f in term 0.
I20260926 12:37:29.499145 29505 leader_election.cc:302] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b52fec9cc354413af981dd513fab89f, 1edcb57dc787409fbb730c92aaf0e86e; no voters:
I20260926 12:37:29.499457 29770 raft_consensus.cc:2781] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:37:29.499552 29770 raft_consensus.cc:482] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:37:29.499601 29770 raft_consensus.cc:3037] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:29.500612 29770 raft_consensus.cc:504] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edcb57dc787409fbb730c92aaf0e86e" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34833 } } peers { permanent_uuid: "cebfe71f687840278e5d42cb21930c47" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 45825 } } peers { permanent_uuid: "1b52fec9cc354413af981dd513fab89f" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 43795 } }
I20260926 12:37:29.500931 29770 leader_election.cc:288] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f [CANDIDATE]: Term 1 election: Requested vote from peers 1edcb57dc787409fbb730c92aaf0e86e (127.10.98.1:34833), cebfe71f687840278e5d42cb21930c47 (127.10.98.3:45825)
I20260926 12:37:29.501300 29439 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27450662feaf4d47ba09c9d9d45ef075" candidate_uuid: "1b52fec9cc354413af981dd513fab89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1edcb57dc787409fbb730c92aaf0e86e"
I20260926 12:37:29.501355 29701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27450662feaf4d47ba09c9d9d45ef075" candidate_uuid: "1b52fec9cc354413af981dd513fab89f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cebfe71f687840278e5d42cb21930c47"
I20260926 12:37:29.501466 29701 raft_consensus.cc:3037] T 27450662feaf4d47ba09c9d9d45ef075 P cebfe71f687840278e5d42cb21930c47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:29.501433 29439 raft_consensus.cc:3037] T 27450662feaf4d47ba09c9d9d45ef075 P 1edcb57dc787409fbb730c92aaf0e86e [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:29.502712 29439 raft_consensus.cc:2454] T 27450662feaf4d47ba09c9d9d45ef075 P 1edcb57dc787409fbb730c92aaf0e86e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b52fec9cc354413af981dd513fab89f in term 1.
I20260926 12:37:29.502712 29701 raft_consensus.cc:2454] T 27450662feaf4d47ba09c9d9d45ef075 P cebfe71f687840278e5d42cb21930c47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b52fec9cc354413af981dd513fab89f in term 1.
I20260926 12:37:29.503016 29505 leader_election.cc:302] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b52fec9cc354413af981dd513fab89f, 1edcb57dc787409fbb730c92aaf0e86e; no voters:
I20260926 12:37:29.503188 29770 raft_consensus.cc:2781] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:29.503400 29770 raft_consensus.cc:686] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f [term 1 LEADER]: Becoming Leader. State: Replica: 1b52fec9cc354413af981dd513fab89f, State: Running, Role: LEADER
I20260926 12:37:29.503750 29770 consensus_queue.cc:237] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edcb57dc787409fbb730c92aaf0e86e" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34833 } } peers { permanent_uuid: "cebfe71f687840278e5d42cb21930c47" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 45825 } } peers { permanent_uuid: "1b52fec9cc354413af981dd513fab89f" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 43795 } }
I20260926 12:37:29.507002 29295 catalog_manager.cc:5495] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b52fec9cc354413af981dd513fab89f (127.10.98.2). New cstate: current_term: 1 leader_uuid: "1b52fec9cc354413af981dd513fab89f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1edcb57dc787409fbb730c92aaf0e86e" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cebfe71f687840278e5d42cb21930c47" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 45825 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b52fec9cc354413af981dd513fab89f" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 43795 } health_report { overall_health: HEALTHY } } }
I20260926 12:37:29.544377 29762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:29.540833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37660 (local address 127.10.98.2:43795)
0926 12:37:29.540953 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:29.540957 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:29.540970 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:37:29.541075 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:29.541077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:29.541160 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:29.541267 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:37:29.541540 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.542041 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.542917 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:29.543144 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:29.543225 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:29.543525 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:37:29.543938 (+ 413us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:37:29.544044 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:29.544195 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 12:37:29.544228 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 12:37:29.550259 29701 raft_consensus.cc:1261] T 27450662feaf4d47ba09c9d9d45ef075 P cebfe71f687840278e5d42cb21930c47 [term 1 FOLLOWER]: Refusing update from remote peer 1b52fec9cc354413af981dd513fab89f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:37:29.550294 29439 raft_consensus.cc:1261] T 27450662feaf4d47ba09c9d9d45ef075 P 1edcb57dc787409fbb730c92aaf0e86e [term 1 FOLLOWER]: Refusing update from remote peer 1b52fec9cc354413af981dd513fab89f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:37:29.550946 29770 consensus_queue.cc:1035] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1edcb57dc787409fbb730c92aaf0e86e" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:37:29.551148 29774 consensus_queue.cc:1035] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cebfe71f687840278e5d42cb21930c47" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 45825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:37:29.559536 29778 mvcc.cc:205] Tried to move back new op lower bound from 7333585918150643712 to 7333585917970100224. Current Snapshot: MvccSnapshot[applied={T|T < 7333585918150643712}]
I20260926 12:37:29.564038 29780 mvcc.cc:205] Tried to move back new op lower bound from 7333585918150643712 to 7333585917970100224. Current Snapshot: MvccSnapshot[applied={T|T < 7333585918150643712}]
I20260926 12:37:29.568898 29295 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:37:29.569046 29295 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55848:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:37:29.571441 29295 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b5d8685d978a41159a8aa4f260c2bff4: Sending DeleteTablet for 3 replicas of tablet 27450662feaf4d47ba09c9d9d45ef075
I20260926 12:37:29.572324 29419 tablet_service.cc:1514] Processing DeleteTablet for tablet 27450662feaf4d47ba09c9d9d45ef075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:37:29 UTC) from {username='slave'} at 127.0.0.1:46136
I20260926 12:37:29.572324 29550 tablet_service.cc:1514] Processing DeleteTablet for tablet 27450662feaf4d47ba09c9d9d45ef075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:37:29 UTC) from {username='slave'} at 127.0.0.1:37658
I20260926 12:37:29.572865 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 29357
I20260926 12:37:29.572990 29681 tablet_service.cc:1514] Processing DeleteTablet for tablet 27450662feaf4d47ba09c9d9d45ef075 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:37:29 UTC) from {username='slave'} at 127.0.0.1:36180
I20260926 12:37:29.573638 29788 tablet_replica.cc:330] T 27450662feaf4d47ba09c9d9d45ef075 P cebfe71f687840278e5d42cb21930c47: stopping tablet replica
I20260926 12:37:29.573936 29788 raft_consensus.cc:2229] T 27450662feaf4d47ba09c9d9d45ef075 P cebfe71f687840278e5d42cb21930c47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:37:29.574222 29788 raft_consensus.cc:2258] T 27450662feaf4d47ba09c9d9d45ef075 P cebfe71f687840278e5d42cb21930c47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:37:29.575416 29788 ts_tablet_manager.cc:1905] T 27450662feaf4d47ba09c9d9d45ef075 P cebfe71f687840278e5d42cb21930c47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:37:29.577323 29786 tablet_replica.cc:330] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f: stopping tablet replica
I20260926 12:37:29.577643 29786 raft_consensus.cc:2229] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:37:29.578028 29786 raft_consensus.cc:2258] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:37:29.578195 29788 ts_tablet_manager.cc:1918] T 27450662feaf4d47ba09c9d9d45ef075 P cebfe71f687840278e5d42cb21930c47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:37:29.578284 29788 log.cc:1196] T 27450662feaf4d47ba09c9d9d45ef075 P cebfe71f687840278e5d42cb21930c47: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/27450662feaf4d47ba09c9d9d45ef075
I20260926 12:37:29.578563 29788 ts_tablet_manager.cc:1939] T 27450662feaf4d47ba09c9d9d45ef075 P cebfe71f687840278e5d42cb21930c47: Deleting consensus metadata
I20260926 12:37:29.579725 29786 ts_tablet_manager.cc:1905] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:37:29.579777 29283 catalog_manager.cc:4841] TS cebfe71f687840278e5d42cb21930c47 (127.10.98.3:45825): tablet 27450662feaf4d47ba09c9d9d45ef075 (table test-table [id=0dd47359e3e940a3ae68d7b1e2e312c6]) successfully deleted
I20260926 12:37:29.582206 29786 ts_tablet_manager.cc:1918] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:37:29.582293 29786 log.cc:1196] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/27450662feaf4d47ba09c9d9d45ef075
I20260926 12:37:29.582540 29786 ts_tablet_manager.cc:1939] T 27450662feaf4d47ba09c9d9d45ef075 P 1b52fec9cc354413af981dd513fab89f: Deleting consensus metadata
W20260926 12:37:29.583345 29283 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.98.1:34833 (error 108)
I20260926 12:37:29.583819 29281 catalog_manager.cc:4841] TS 1b52fec9cc354413af981dd513fab89f (127.10.98.2:43795): tablet 27450662feaf4d47ba09c9d9d45ef075 (table test-table [id=0dd47359e3e940a3ae68d7b1e2e312c6]) successfully deleted
I20260926 12:37:29.584110 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 29488
I20260926 12:37:29.584426 29761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:29.584148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:34833 (local address 127.0.0.1:46148)
0926 12:37:29.584279 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:29.584342 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.1:34833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260926 12:37:29.584652 29283 catalog_manager.cc:4584] TS 1edcb57dc787409fbb730c92aaf0e86e (127.10.98.1:34833): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27450662feaf4d47ba09c9d9d45ef075: Network error: Client connection negotiation failed: client connection to 127.10.98.1:34833: connect: Connection refused (error 111)
I20260926 12:37:29.589600 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 29619
I20260926 12:37:29.594203 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 29266
2026-09-26T12:37:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1072 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:29 dist-test-slave-t7tk krb5kdc[29792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:29 dist-test-slave-t7tk krb5kdc[29792](info): set up 2 sockets
Sep 26 12:37:29 dist-test-slave-t7tk krb5kdc[29792](info): commencing operation
krb5kdc: starting...
W20260926 12:37:30.554388 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:30 dist-test-slave-t7tk krb5kdc[29792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:37:30.673939 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:43859
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:46325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:43859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:30.783852 29805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:30.784138 29805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:30.784178 29805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:30.788676 29805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:30.788744 29805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:30.788765 29805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:30.788789 29805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:30.788807 29805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:30.794302 29805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:46325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:43859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:43859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:30.795354 29805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:30.796339 29805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:30.805032 29813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:30.805039 29811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:30.805069 29810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:30.805089 29805 server_base.cc:1034] running on GCE node
I20260926 12:37:30.805788 29805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:30.806917 29805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:30.808110 29805 hybrid_clock.cc:648] HybridClock initialized: now 1790426250808081 us; error 33 us; skew 500 ppm
Sep 26 12:37:30 dist-test-slave-t7tk krb5kdc[29792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426250, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:30.810995 29805 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:37:30.811915 29805 webserver.cc:460] Webserver started at http://127.10.98.62:33105/ using document root <none> and password file <none>
I20260926 12:37:30.812480 29805 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:30.812558 29805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:30.812754 29805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:30.814888 29805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "73b769a9896c4e7bb413b4af6c9556d4"
format_stamp: "Formatted at 2026-09-26 12:37:30 on dist-test-slave-t7tk"
server_key: "d2bba0668165f3dc327c03c2023a2589"
server_key_iv: "4ebf4d68b2ad49d33677f4e479eeb7e2"
server_key_version: "encryptionkey@0"
I20260926 12:37:30.815428 29805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "73b769a9896c4e7bb413b4af6c9556d4"
format_stamp: "Formatted at 2026-09-26 12:37:30 on dist-test-slave-t7tk"
server_key: "d2bba0668165f3dc327c03c2023a2589"
server_key_iv: "4ebf4d68b2ad49d33677f4e479eeb7e2"
server_key_version: "encryptionkey@0"
I20260926 12:37:30.819164 29805 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:37:30.821707 29820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:30.823146 29805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:37:30.823271 29805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "73b769a9896c4e7bb413b4af6c9556d4"
format_stamp: "Formatted at 2026-09-26 12:37:30 on dist-test-slave-t7tk"
server_key: "d2bba0668165f3dc327c03c2023a2589"
server_key_iv: "4ebf4d68b2ad49d33677f4e479eeb7e2"
server_key_version: "encryptionkey@0"
I20260926 12:37:30.823388 29805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:30.845479 29805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:30.848726 29805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:30.848917 29805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:30.855919 29805 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:43859
I20260926 12:37:30.855940 29872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:43859 every 8 connection(s)
I20260926 12:37:30.856933 29805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:30.859679 29873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:30.860065 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 29805
I20260926 12:37:30.860200 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:30.860535 10632 external_mini_cluster.cc:1428] Setting key f8918a4cab4fd9f6185629e828100fa3
I20260926 12:37:30.866894 29873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4: Bootstrap starting.
I20260926 12:37:30.869586 29873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4: Neither blocks nor log segments found. Creating new log.
Sep 26 12:37:30 dist-test-slave-t7tk krb5kdc[29792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:37:30.870329 29873 log.cc:824] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:30.872367 29873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4: No bootstrap required, opened a new log
I20260926 12:37:30.875054 29876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:30.861921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56398 (local address 127.10.98.62:43859)
0926 12:37:30.862311 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:30.862328 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:30.862372 (+ 44us) server_negotiation.cc:407] Connection header received
0926 12:37:30.863066 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:30.863085 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:30.863425 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:30.863701 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:30.864600 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:30.865744 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:30.866923 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:30.867526 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:30.867653 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:30.870484 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:30.870491 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:30.870509 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:30.870542 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:30.872239 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:30.872737 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:30.872742 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:30.872746 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:30.872858 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:30.873134 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:30.873137 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:30.873139 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:30.873551 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:30.873725 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:30.874040 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 12:37:30.874316 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":114,"threads_started":1}
I20260926 12:37:30.876039 29873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b769a9896c4e7bb413b4af6c9556d4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43859 } }
I20260926 12:37:30.876282 29873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:30.876339 29873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73b769a9896c4e7bb413b4af6c9556d4, State: Initialized, Role: FOLLOWER
I20260926 12:37:30.876749 29873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b769a9896c4e7bb413b4af6c9556d4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43859 } }
I20260926 12:37:30.876879 29873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:30.876945 29873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:30.877040 29873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:30.877981 29873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b769a9896c4e7bb413b4af6c9556d4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43859 } }
I20260926 12:37:30.878274 29873 leader_election.cc:302] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73b769a9896c4e7bb413b4af6c9556d4; no voters:
I20260926 12:37:30.878551 29873 leader_election.cc:288] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:30.878654 29878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:30.878839 29878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4 [term 1 LEADER]: Becoming Leader. State: Replica: 73b769a9896c4e7bb413b4af6c9556d4, State: Running, Role: LEADER
I20260926 12:37:30.879199 29878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b769a9896c4e7bb413b4af6c9556d4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43859 } }
I20260926 12:37:30.879819 29873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:30.880898 29880 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73b769a9896c4e7bb413b4af6c9556d4. Latest consensus state: current_term: 1 leader_uuid: "73b769a9896c4e7bb413b4af6c9556d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b769a9896c4e7bb413b4af6c9556d4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43859 } } }
I20260926 12:37:30.881026 29880 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:30.881387 29879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73b769a9896c4e7bb413b4af6c9556d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73b769a9896c4e7bb413b4af6c9556d4" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43859 } } }
I20260926 12:37:30.881466 29879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:30.881531 29887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:30.884660 29887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:30.890408 29887 catalog_manager.cc:1347] Generated new cluster ID: a47c016ac0e24abab569d6c3906c1889
I20260926 12:37:30.890484 29887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:30.906168 29887 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:30.907295 29887 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:30.915720 29887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4: Generated new TSK 0
I20260926 12:37:30.916505 29887 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.10.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
Entry for principal kudu/127.10.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab.
I20260926 12:37:30.951699 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:43859
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:46325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:31.061061 29899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:31.061369 29899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:31.061429 29899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:31.065516 29899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:31.065589 29899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:31.065672 29899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:37:31.071077 29899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:46325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.1.keytab
--principal=kudu/127.10.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:43859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:31.072170 29899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:31.073231 29899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:31.083212 29905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:31.083177 29904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:31.083184 29907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:31.083299 29899 server_base.cc:1034] running on GCE node
I20260926 12:37:31.083910 29899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:31.084568 29899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:31.085717 29899 hybrid_clock.cc:648] HybridClock initialized: now 1790426251085699 us; error 34 us; skew 500 ppm
Sep 26 12:37:31 dist-test-slave-t7tk krb5kdc[29792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426251, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:31.088901 29899 init.cc:377] Logged in from keytab as kudu/127.10.98.1@KRBTEST.COM (short username kudu)
I20260926 12:37:31.089871 29899 webserver.cc:460] Webserver started at http://127.10.98.1:38119/ using document root <none> and password file <none>
I20260926 12:37:31.090435 29899 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:31.090502 29899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:31.090696 29899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:31.092783 29899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "1c29740e9a454f18b647ca7f6e6f98f2"
format_stamp: "Formatted at 2026-09-26 12:37:31 on dist-test-slave-t7tk"
server_key: "541e10b77e6ff00bc51c6acf2d1c0add"
server_key_iv: "a7f26c02293ef8d8ca699200255def95"
server_key_version: "encryptionkey@0"
I20260926 12:37:31.093387 29899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "1c29740e9a454f18b647ca7f6e6f98f2"
format_stamp: "Formatted at 2026-09-26 12:37:31 on dist-test-slave-t7tk"
server_key: "541e10b77e6ff00bc51c6acf2d1c0add"
server_key_iv: "a7f26c02293ef8d8ca699200255def95"
server_key_version: "encryptionkey@0"
I20260926 12:37:31.097448 29899 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260926 12:37:31.100147 29914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:31.101487 29899 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:37:31.101593 29899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "1c29740e9a454f18b647ca7f6e6f98f2"
format_stamp: "Formatted at 2026-09-26 12:37:31 on dist-test-slave-t7tk"
server_key: "541e10b77e6ff00bc51c6acf2d1c0add"
server_key_iv: "a7f26c02293ef8d8ca699200255def95"
server_key_version: "encryptionkey@0"
I20260926 12:37:31.101689 29899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:31.127259 29899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:31.130475 29899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:31.130642 29899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:31.131203 29899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:37:31.132002 29899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:37:31.132057 29899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:31.132098 29899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:37:31.132125 29899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:31.140618 29899 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:39677
I20260926 12:37:31.140748 30027 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:39677 every 8 connection(s)
I20260926 12:37:31.141572 29899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:37:31.148408 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 29899
I20260926 12:37:31.148571 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:37:31.148876 10632 external_mini_cluster.cc:1428] Setting key 7e343a9d5445da21ef3640e5073620f7
Sep 26 12:37:31 dist-test-slave-t7tk krb5kdc[29792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426251, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.1@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:37:31.155678 29876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:31.143655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:50985 (local address 127.10.98.62:43859)
0926 12:37:31.143817 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:31.143820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:31.144581 (+ 761us) server_negotiation.cc:407] Connection header received
0926 12:37:31.145909 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:31.145912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:31.145958 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:31.146034 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:31.147249 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.147785 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.149338 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.149639 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.149734 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:31.152170 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:31.152173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:31.152176 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:31.152203 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:31.153651 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:31.154143 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:31.154146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:31.154148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:31.154231 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:31.154592 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:31.154595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:31.154596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:31.154776 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:31.154861 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:31.155371 (+ 510us) server_negotiation.cc:299] Negotiation successful
0926 12:37:31.155503 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 12:37:31.156513 30030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:31.143951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:43859 (local address 127.10.98.1:50985)
0926 12:37:31.144418 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:31.144463 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:31.145706 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:31.146199 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:31.146207 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:31.146595 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:31.147103 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:31.147119 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.147902 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.147905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:31.149191 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:31.149203 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.149815 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.149820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:31.149997 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:31.150570 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:37:31.150586 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:31.152003 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:31.153760 (+ 1757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:31.153766 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:31.153783 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:31.154049 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:31.154312 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:31.154315 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:31.154317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:31.154460 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:31.154876 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:31.154884 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:31.155143 (+ 259us) client_negotiation.cc:769] Sending connection context
0926 12:37:31.155313 (+ 170us) client_negotiation.cc:240] Negotiation successful
0926 12:37:31.155658 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":118,"threads_started":1}
I20260926 12:37:31.157795 30028 heartbeater.cc:347] Connected to a master server at 127.10.98.62:43859
WARNING: no policy specified for kudu/127.10.98.2@KRBTEST.COM; defaulting to no policy
I20260926 12:37:31.158025 30028 heartbeater.cc:464] Registering TS with master...
I20260926 12:37:31.158507 30028 heartbeater.cc:511] Master 127.10.98.62:43859 requested a full tablet report, sending...
I20260926 12:37:31.159996 29837 ts_manager.cc:194] Registered new tserver with Master: 1c29740e9a454f18b647ca7f6e6f98f2 (127.10.98.1:39677)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.2@KRBTEST.COM" created.
I20260926 12:37:31.161429 29837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.1@KRBTEST.COM'} at 127.10.98.1:50985
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
Entry for principal kudu/127.10.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab.
I20260926 12:37:31.177346 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:43859
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:46325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:31.287565 30033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:31.287832 30033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:31.287873 30033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:31.291951 30033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:31.292018 30033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:31.292085 30033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:37:31.297406 30033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:46325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.2.keytab
--principal=kudu/127.10.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:43859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:31.298573 30033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:31.299600 30033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:31.309258 30041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:31.309304 30033 server_base.cc:1034] running on GCE node
W20260926 12:37:31.309252 30038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:31.309269 30039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:31.309947 30033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:31.310598 30033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:31.311852 30033 hybrid_clock.cc:648] HybridClock initialized: now 1790426251311834 us; error 31 us; skew 500 ppm
Sep 26 12:37:31 dist-test-slave-t7tk krb5kdc[29792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426251, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:31.314787 30033 init.cc:377] Logged in from keytab as kudu/127.10.98.2@KRBTEST.COM (short username kudu)
I20260926 12:37:31.315737 30033 webserver.cc:460] Webserver started at http://127.10.98.2:43895/ using document root <none> and password file <none>
I20260926 12:37:31.316308 30033 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:31.316372 30033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:31.316550 30033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:31.318640 30033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "a5f726dd76bc4159b49e063bad3d9e1b"
format_stamp: "Formatted at 2026-09-26 12:37:31 on dist-test-slave-t7tk"
server_key: "cf3114084669b5e89904db535b853805"
server_key_iv: "c64c77f9ab9a4c93379a3a9ff518207e"
server_key_version: "encryptionkey@0"
I20260926 12:37:31.319164 30033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "a5f726dd76bc4159b49e063bad3d9e1b"
format_stamp: "Formatted at 2026-09-26 12:37:31 on dist-test-slave-t7tk"
server_key: "cf3114084669b5e89904db535b853805"
server_key_iv: "c64c77f9ab9a4c93379a3a9ff518207e"
server_key_version: "encryptionkey@0"
I20260926 12:37:31.322975 30033 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:37:31.325659 30048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:31.327066 30033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:37:31.327200 30033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "a5f726dd76bc4159b49e063bad3d9e1b"
format_stamp: "Formatted at 2026-09-26 12:37:31 on dist-test-slave-t7tk"
server_key: "cf3114084669b5e89904db535b853805"
server_key_iv: "c64c77f9ab9a4c93379a3a9ff518207e"
server_key_version: "encryptionkey@0"
I20260926 12:37:31.327299 30033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:31.347635 30033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:31.350831 30033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:31.351029 30033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:31.351635 30033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:37:31.352463 30033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:37:31.352507 30033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:31.352560 30033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:37:31.352591 30033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:31.360957 30033 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:37397
I20260926 12:37:31.361050 30161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:37397 every 8 connection(s)
I20260926 12:37:31.361903 30033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:37:31.363888 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 30033
I20260926 12:37:31.364027 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:37:31.364303 10632 external_mini_cluster.cc:1428] Setting key e51b3e226c439fc2b32ef17971af122f
Sep 26 12:37:31 dist-test-slave-t7tk krb5kdc[29792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426251, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.2@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.10.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.3@KRBTEST.COM" created.
I20260926 12:37:31.375887 29876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:31.363731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:60177 (local address 127.10.98.62:43859)
0926 12:37:31.363893 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:31.363898 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:31.364780 (+ 882us) server_negotiation.cc:407] Connection header received
0926 12:37:31.366101 (+ 1321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:31.366104 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:31.366153 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:31.366231 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:31.367453 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.368236 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.369420 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.369655 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.369772 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:31.372387 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:31.372390 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:31.372392 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:31.372417 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:31.373762 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:31.374308 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:31.374312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:31.374314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:31.374389 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:31.374711 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:31.374714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:31.374716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:31.374934 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:31.375018 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:31.375593 (+ 575us) server_negotiation.cc:299] Negotiation successful
0926 12:37:31.375727 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 12:37:31.376726 30164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:31.364131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:43859 (local address 127.10.98.2:60177)
0926 12:37:31.364614 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:31.364669 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:31.365892 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:31.366414 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:31.366421 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:31.366817 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:31.367310 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:31.367323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.368365 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.368368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:31.369305 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:31.369314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.369824 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.369828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:31.369993 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:31.370564 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:37:31.370582 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:31.372217 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:31.373900 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:31.373906 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:31.373923 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:31.374212 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:31.374470 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:31.374473 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:31.374475 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:31.374621 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:31.375048 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:31.375055 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:31.375322 (+ 267us) client_negotiation.cc:769] Sending connection context
0926 12:37:31.375538 (+ 216us) client_negotiation.cc:240] Negotiation successful
0926 12:37:31.375883 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":116,"threads_started":1}
I20260926 12:37:31.377872 30162 heartbeater.cc:347] Connected to a master server at 127.10.98.62:43859
I20260926 12:37:31.378103 30162 heartbeater.cc:464] Registering TS with master...
I20260926 12:37:31.378584 30162 heartbeater.cc:511] Master 127.10.98.62:43859 requested a full tablet report, sending...
I20260926 12:37:31.379628 29837 ts_manager.cc:194] Registered new tserver with Master: a5f726dd76bc4159b49e063bad3d9e1b (127.10.98.2:37397)
I20260926 12:37:31.380419 29837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.2@KRBTEST.COM'} at 127.10.98.2:60177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
Entry for principal kudu/127.10.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab.
I20260926 12:37:31.392680 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:43859
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:46325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:31.503269 30167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:31.503551 30167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:31.503593 30167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:31.507908 30167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:31.507970 30167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:31.508031 30167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:37:31.513408 30167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:46325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.3.keytab
--principal=kudu/127.10.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:43859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:31.514448 30167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:31.515411 30167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:31.525022 30172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:31.525128 30175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:31.525053 30173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:31.525168 30167 server_base.cc:1034] running on GCE node
I20260926 12:37:31.525702 30167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:31.526335 30167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:31.527518 30167 hybrid_clock.cc:648] HybridClock initialized: now 1790426251527493 us; error 32 us; skew 500 ppm
Sep 26 12:37:31 dist-test-slave-t7tk krb5kdc[29792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426251, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:31.530282 30167 init.cc:377] Logged in from keytab as kudu/127.10.98.3@KRBTEST.COM (short username kudu)
I20260926 12:37:31.531173 30167 webserver.cc:460] Webserver started at http://127.10.98.3:43315/ using document root <none> and password file <none>
I20260926 12:37:31.531723 30167 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:31.531795 30167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:31.531996 30167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:31.534075 30167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "e86a1f30ca2a45cca27cb5a9c5b57fed"
format_stamp: "Formatted at 2026-09-26 12:37:31 on dist-test-slave-t7tk"
server_key: "22e1f3723e4de92fd2ea5477f15c015a"
server_key_iv: "e31125a357287b7554daae4c9910fe77"
server_key_version: "encryptionkey@0"
I20260926 12:37:31.534610 30167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "e86a1f30ca2a45cca27cb5a9c5b57fed"
format_stamp: "Formatted at 2026-09-26 12:37:31 on dist-test-slave-t7tk"
server_key: "22e1f3723e4de92fd2ea5477f15c015a"
server_key_iv: "e31125a357287b7554daae4c9910fe77"
server_key_version: "encryptionkey@0"
I20260926 12:37:31.538306 30167 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260926 12:37:31.540886 30182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:31.542232 30167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:37:31.542343 30167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "e86a1f30ca2a45cca27cb5a9c5b57fed"
format_stamp: "Formatted at 2026-09-26 12:37:31 on dist-test-slave-t7tk"
server_key: "22e1f3723e4de92fd2ea5477f15c015a"
server_key_iv: "e31125a357287b7554daae4c9910fe77"
server_key_version: "encryptionkey@0"
I20260926 12:37:31.542444 30167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:31.555481 30167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:31.559000 30167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:31.559187 30167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:31.559747 30167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:37:31.560560 30167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:37:31.560607 30167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:31.560660 30167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:37:31.560688 30167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:31.568856 30167 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:36527
I20260926 12:37:31.568887 30295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:36527 every 8 connection(s)
I20260926 12:37:31.569808 30167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:37:31.578965 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 30167
Sep 26 12:37:31 dist-test-slave-t7tk krb5kdc[29792](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426251, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.3@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:37:31.579114 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:37:31.579422 10632 external_mini_cluster.cc:1428] Setting key 08cbd9581467c305f8c07e5ddb762b70
I20260926 12:37:31.583235 29876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:31.571501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:36157 (local address 127.10.98.62:43859)
0926 12:37:31.571673 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:31.571676 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:31.572437 (+ 761us) server_negotiation.cc:407] Connection header received
0926 12:37:31.573774 (+ 1337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:31.573778 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:31.573835 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:31.573908 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:31.575072 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.575608 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.576819 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.577038 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.577164 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:31.579818 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:31.579822 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:31.579825 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:31.579856 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:31.581401 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:31.581912 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:31.581915 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:31.581916 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:31.581992 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:31.582272 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:31.582274 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:31.582275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:31.582442 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:31.582513 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:31.582991 (+ 478us) server_negotiation.cc:299] Negotiation successful
0926 12:37:31.583115 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260926 12:37:31.584054 30298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:31.571859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:43859 (local address 127.10.98.3:36157)
0926 12:37:31.572293 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:31.572335 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:31.573558 (+ 1223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:31.574097 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:31.574104 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:31.574457 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:31.574930 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:31.574942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.575752 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.575755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:31.576699 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:31.576709 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.577208 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.577212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:31.577377 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:31.577904 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:37:31.577921 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:31.579632 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:31.581508 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:31.581514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:31.581532 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:31.581834 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:31.582081 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:31.582083 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:31.582084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:31.582211 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:31.582512 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:31.582517 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:31.582792 (+ 275us) client_negotiation.cc:769] Sending connection context
0926 12:37:31.582952 (+ 160us) client_negotiation.cc:240] Negotiation successful
0926 12:37:31.583229 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":121,"threads_started":1}
I20260926 12:37:31.585127 30296 heartbeater.cc:347] Connected to a master server at 127.10.98.62:43859
I20260926 12:37:31.585352 30296 heartbeater.cc:464] Registering TS with master...
I20260926 12:37:31.585799 30296 heartbeater.cc:511] Master 127.10.98.62:43859 requested a full tablet report, sending...
I20260926 12:37:31.586760 29837 ts_manager.cc:194] Registered new tserver with Master: e86a1f30ca2a45cca27cb5a9c5b57fed (127.10.98.3:36527)
I20260926 12:37:31.587312 29837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.10.98.3@KRBTEST.COM'} at 127.10.98.3:36157
I20260926 12:37:31.593345 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:37:31.602149 29876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:31.595204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56408 (local address 127.10.98.62:43859)
0926 12:37:31.595334 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:31.595339 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:31.595519 (+ 180us) server_negotiation.cc:407] Connection header received
0926 12:37:31.595638 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:31.595641 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:31.595687 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:31.595753 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:31.596446 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.596959 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.597810 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.598032 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.598118 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:31.599331 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:31.599335 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:31.599338 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:31.599364 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:31.600512 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:31.600931 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:31.600935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:31.600936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:31.601015 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:31.601283 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:31.601286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:31.601288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:31.601507 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:31.601631 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:31.601863 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 12:37:31.601974 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:37:31.604946 29837 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56408:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:37:31.608028 29837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:37:31.622315 30308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:31.616483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:39677 (local address 127.0.0.1:42458)
0926 12:37:31.617527 (+ 1044us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:31.617540 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:31.617623 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:31.618491 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:31.618494 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:31.618512 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:31.618636 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:31.618642 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.619595 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.619599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:31.620804 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:31.620813 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.621790 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.621793 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:31.621899 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:31.621932 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 12:37:31.622045 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 12:37:31.622165 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"mutex_wait_us":37,"thread_start_us":109,"threads_started":1}
I20260926 12:37:31.622329 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:31.617197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42458 (local address 127.10.98.1:39677)
0926 12:37:31.618081 (+ 884us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:31.618095 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:31.618114 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:37:31.618180 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:31.618184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:31.618340 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:31.618507 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:31.618740 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.619469 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.620920 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.621637 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.621742 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:31.621887 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:31.622101 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 12:37:31.622194 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":73,"threads_started":1}
I20260926 12:37:31.622867 30310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:31.616672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:37397 (local address 127.0.0.1:36928)
0926 12:37:31.617758 (+ 1086us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:31.617768 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:31.617853 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:31.618394 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:31.618397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:31.618413 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:31.618506 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:31.618512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.619877 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.619880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:31.621250 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:31.621256 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.622455 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.622458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:31.622569 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:31.622594 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 12:37:31.622670 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 12:37:31.622739 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1040,"mutex_wait_us":124,"thread_start_us":84,"threads_started":1}
I20260926 12:37:31.622969 30312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:31.617163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36928 (local address 127.10.98.2:37397)
0926 12:37:31.617644 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:31.617656 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:31.617849 (+ 193us) server_negotiation.cc:407] Connection header received
0926 12:37:31.617956 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:31.617962 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:31.618160 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:31.618299 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:31.618624 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.619728 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.621376 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.622167 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.622266 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:31.622395 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:31.622727 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 12:37:31.622831 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":141,"threads_started":1}
I20260926 12:37:31.623804 30307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:31.616343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:36527 (local address 127.0.0.1:49490)
0926 12:37:31.617158 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:31.617207 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:31.617333 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:31.618227 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:31.618233 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:31.618253 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:31.618378 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:31.618385 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.619877 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.619880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:31.621236 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:31.621248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.623470 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.623473 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:31.623560 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:31.623583 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 12:37:31.623640 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 12:37:31.623709 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"mutex_wait_us":50,"thread_start_us":80,"threads_started":1}
I20260926 12:37:31.624399 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:31.616495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49490 (local address 127.10.98.3:36527)
0926 12:37:31.617829 (+ 1334us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:31.617837 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:31.617854 (+ 17us) server_negotiation.cc:407] Connection header received
0926 12:37:31.617914 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:31.617918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:31.618086 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:31.618220 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:31.618502 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.619725 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.621965 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.622773 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.623939 (+ 1166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:31.624099 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:31.624152 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 12:37:31.624243 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1249,"thread_start_us":145,"threads_started":1}
I20260926 12:37:31.625800 30096 tablet_service.cc:1467] Processing CreateTablet for tablet 43e78ea4a6964b85b9ba4fa1e15ebfe4 (DEFAULT_TABLE table=test-table [id=2b23a7bcb1f440afa55ed04ac8715180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:37:31.625772 29962 tablet_service.cc:1467] Processing CreateTablet for tablet 43e78ea4a6964b85b9ba4fa1e15ebfe4 (DEFAULT_TABLE table=test-table [id=2b23a7bcb1f440afa55ed04ac8715180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:37:31.626920 29962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e78ea4a6964b85b9ba4fa1e15ebfe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:31.626917 30096 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e78ea4a6964b85b9ba4fa1e15ebfe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:31.627313 30230 tablet_service.cc:1467] Processing CreateTablet for tablet 43e78ea4a6964b85b9ba4fa1e15ebfe4 (DEFAULT_TABLE table=test-table [id=2b23a7bcb1f440afa55ed04ac8715180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:37:31.628445 30230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43e78ea4a6964b85b9ba4fa1e15ebfe4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:31.633193 30313 tablet_bootstrap.cc:492] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b: Bootstrap starting.
I20260926 12:37:31.635414 30314 tablet_bootstrap.cc:492] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P 1c29740e9a454f18b647ca7f6e6f98f2: Bootstrap starting.
I20260926 12:37:31.635839 30313 tablet_bootstrap.cc:654] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:31.636000 30315 tablet_bootstrap.cc:492] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P e86a1f30ca2a45cca27cb5a9c5b57fed: Bootstrap starting.
I20260926 12:37:31.636575 30313 log.cc:824] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:31.637631 30314 tablet_bootstrap.cc:654] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P 1c29740e9a454f18b647ca7f6e6f98f2: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:31.638079 30315 tablet_bootstrap.cc:654] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P e86a1f30ca2a45cca27cb5a9c5b57fed: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:31.638747 30313 tablet_bootstrap.cc:492] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b: No bootstrap required, opened a new log
I20260926 12:37:31.638886 30314 log.cc:824] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P 1c29740e9a454f18b647ca7f6e6f98f2: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:31.638993 30313 ts_tablet_manager.cc:1397] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:37:31.639304 30315 log.cc:824] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P e86a1f30ca2a45cca27cb5a9c5b57fed: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:31.640589 30314 tablet_bootstrap.cc:492] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P 1c29740e9a454f18b647ca7f6e6f98f2: No bootstrap required, opened a new log
I20260926 12:37:31.640749 30314 ts_tablet_manager.cc:1397] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P 1c29740e9a454f18b647ca7f6e6f98f2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:37:31.642284 30313 raft_consensus.cc:348] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86a1f30ca2a45cca27cb5a9c5b57fed" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36527 } } peers { permanent_uuid: "1c29740e9a454f18b647ca7f6e6f98f2" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 39677 } } peers { permanent_uuid: "a5f726dd76bc4159b49e063bad3d9e1b" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37397 } }
I20260926 12:37:31.642598 30313 raft_consensus.cc:374] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:31.642681 30313 raft_consensus.cc:729] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5f726dd76bc4159b49e063bad3d9e1b, State: Initialized, Role: FOLLOWER
I20260926 12:37:31.642921 30315 tablet_bootstrap.cc:492] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P e86a1f30ca2a45cca27cb5a9c5b57fed: No bootstrap required, opened a new log
I20260926 12:37:31.643148 30315 ts_tablet_manager.cc:1397] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P e86a1f30ca2a45cca27cb5a9c5b57fed: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260926 12:37:31.643187 30313 consensus_queue.cc:260] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86a1f30ca2a45cca27cb5a9c5b57fed" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36527 } } peers { permanent_uuid: "1c29740e9a454f18b647ca7f6e6f98f2" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 39677 } } peers { permanent_uuid: "a5f726dd76bc4159b49e063bad3d9e1b" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37397 } }
I20260926 12:37:31.643854 30162 heartbeater.cc:503] Master 127.10.98.62:43859 was elected leader, sending a full tablet report...
I20260926 12:37:31.643986 30314 raft_consensus.cc:348] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P 1c29740e9a454f18b647ca7f6e6f98f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86a1f30ca2a45cca27cb5a9c5b57fed" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36527 } } peers { permanent_uuid: "1c29740e9a454f18b647ca7f6e6f98f2" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 39677 } } peers { permanent_uuid: "a5f726dd76bc4159b49e063bad3d9e1b" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37397 } }
I20260926 12:37:31.644203 30313 ts_tablet_manager.cc:1428] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:37:31.644217 30314 raft_consensus.cc:374] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P 1c29740e9a454f18b647ca7f6e6f98f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:31.644263 30314 raft_consensus.cc:729] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P 1c29740e9a454f18b647ca7f6e6f98f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c29740e9a454f18b647ca7f6e6f98f2, State: Initialized, Role: FOLLOWER
I20260926 12:37:31.644699 30314 consensus_queue.cc:260] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P 1c29740e9a454f18b647ca7f6e6f98f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86a1f30ca2a45cca27cb5a9c5b57fed" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36527 } } peers { permanent_uuid: "1c29740e9a454f18b647ca7f6e6f98f2" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 39677 } } peers { permanent_uuid: "a5f726dd76bc4159b49e063bad3d9e1b" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37397 } }
I20260926 12:37:31.645395 30028 heartbeater.cc:503] Master 127.10.98.62:43859 was elected leader, sending a full tablet report...
I20260926 12:37:31.645694 30314 ts_tablet_manager.cc:1428] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P 1c29740e9a454f18b647ca7f6e6f98f2: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:37:31.647739 30315 raft_consensus.cc:348] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P e86a1f30ca2a45cca27cb5a9c5b57fed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86a1f30ca2a45cca27cb5a9c5b57fed" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36527 } } peers { permanent_uuid: "1c29740e9a454f18b647ca7f6e6f98f2" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 39677 } } peers { permanent_uuid: "a5f726dd76bc4159b49e063bad3d9e1b" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37397 } }
I20260926 12:37:31.648020 30315 raft_consensus.cc:374] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P e86a1f30ca2a45cca27cb5a9c5b57fed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:31.648085 30315 raft_consensus.cc:729] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P e86a1f30ca2a45cca27cb5a9c5b57fed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e86a1f30ca2a45cca27cb5a9c5b57fed, State: Initialized, Role: FOLLOWER
I20260926 12:37:31.648499 30315 consensus_queue.cc:260] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P e86a1f30ca2a45cca27cb5a9c5b57fed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86a1f30ca2a45cca27cb5a9c5b57fed" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36527 } } peers { permanent_uuid: "1c29740e9a454f18b647ca7f6e6f98f2" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 39677 } } peers { permanent_uuid: "a5f726dd76bc4159b49e063bad3d9e1b" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37397 } }
I20260926 12:37:31.649139 30296 heartbeater.cc:503] Master 127.10.98.62:43859 was elected leader, sending a full tablet report...
I20260926 12:37:31.649403 30315 ts_tablet_manager.cc:1428] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P e86a1f30ca2a45cca27cb5a9c5b57fed: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:37:31.743621 30319 raft_consensus.cc:482] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:37:31.743837 30319 raft_consensus.cc:504] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86a1f30ca2a45cca27cb5a9c5b57fed" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36527 } } peers { permanent_uuid: "1c29740e9a454f18b647ca7f6e6f98f2" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 39677 } } peers { permanent_uuid: "a5f726dd76bc4159b49e063bad3d9e1b" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37397 } }
I20260926 12:37:31.745056 30319 leader_election.cc:288] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e86a1f30ca2a45cca27cb5a9c5b57fed (127.10.98.3:36527), 1c29740e9a454f18b647ca7f6e6f98f2 (127.10.98.1:39677)
I20260926 12:37:31.749634 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:31.745458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:36383 (local address 127.10.98.1:39677)
0926 12:37:31.745577 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:31.745581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:31.745845 (+ 264us) server_negotiation.cc:407] Connection header received
0926 12:37:31.745997 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:31.746000 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:31.746042 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:31.746106 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:31.746355 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.747099 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.748474 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.749054 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.749179 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:31.749232 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:31.749432 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 12:37:31.749502 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:37:31.749634 30309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:31.745584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:41195 (local address 127.10.98.3:36527)
0926 12:37:31.745681 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:31.745684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:31.745696 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:37:31.745738 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:31.745740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:31.745778 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:31.745838 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:31.746138 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.746867 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.748322 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.748983 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.749092 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:31.749183 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:31.749432 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 12:37:31.749508 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 12:37:31.749648 30164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:31.745286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:36527 (local address 127.10.98.2:41195)
0926 12:37:31.745406 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:31.745421 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:31.745523 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:31.745903 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:31.745905 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:31.745920 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:31.746031 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:31.746036 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.746966 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.746968 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:31.748192 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:31.748203 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.749133 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.749138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:31.749252 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:31.749327 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 12:37:31.749405 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 12:37:31.749496 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":30}
I20260926 12:37:31.749655 30322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:31.745375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:39677 (local address 127.10.98.2:36383)
0926 12:37:31.745757 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:31.745769 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:31.745869 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:31.746127 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:31.746130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:31.746146 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:31.746256 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:31.746262 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.747238 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.747241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:31.748359 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:31.748367 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.749179 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.749182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:31.749332 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:31.749349 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 12:37:31.749410 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 12:37:31.749496 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":47,"thread_start_us":161,"threads_started":1}
I20260926 12:37:31.750388 29982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e78ea4a6964b85b9ba4fa1e15ebfe4" candidate_uuid: "a5f726dd76bc4159b49e063bad3d9e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c29740e9a454f18b647ca7f6e6f98f2" is_pre_election: true
I20260926 12:37:31.750389 30250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e78ea4a6964b85b9ba4fa1e15ebfe4" candidate_uuid: "a5f726dd76bc4159b49e063bad3d9e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e86a1f30ca2a45cca27cb5a9c5b57fed" is_pre_election: true
I20260926 12:37:31.750689 30250 raft_consensus.cc:2454] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P e86a1f30ca2a45cca27cb5a9c5b57fed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f726dd76bc4159b49e063bad3d9e1b in term 0.
I20260926 12:37:31.750690 29982 raft_consensus.cc:2454] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P 1c29740e9a454f18b647ca7f6e6f98f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5f726dd76bc4159b49e063bad3d9e1b in term 0.
I20260926 12:37:31.751155 30051 leader_election.cc:302] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1c29740e9a454f18b647ca7f6e6f98f2, a5f726dd76bc4159b49e063bad3d9e1b, e86a1f30ca2a45cca27cb5a9c5b57fed; no voters:
I20260926 12:37:31.751432 30319 raft_consensus.cc:2781] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:37:31.751514 30319 raft_consensus.cc:482] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:37:31.751606 30319 raft_consensus.cc:3037] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:31.752606 30319 raft_consensus.cc:504] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86a1f30ca2a45cca27cb5a9c5b57fed" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36527 } } peers { permanent_uuid: "1c29740e9a454f18b647ca7f6e6f98f2" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 39677 } } peers { permanent_uuid: "a5f726dd76bc4159b49e063bad3d9e1b" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37397 } }
I20260926 12:37:31.752926 30319 leader_election.cc:288] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b [CANDIDATE]: Term 1 election: Requested vote from peers e86a1f30ca2a45cca27cb5a9c5b57fed (127.10.98.3:36527), 1c29740e9a454f18b647ca7f6e6f98f2 (127.10.98.1:39677)
I20260926 12:37:31.753304 29982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e78ea4a6964b85b9ba4fa1e15ebfe4" candidate_uuid: "a5f726dd76bc4159b49e063bad3d9e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c29740e9a454f18b647ca7f6e6f98f2"
I20260926 12:37:31.753302 30250 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43e78ea4a6964b85b9ba4fa1e15ebfe4" candidate_uuid: "a5f726dd76bc4159b49e063bad3d9e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e86a1f30ca2a45cca27cb5a9c5b57fed"
I20260926 12:37:31.753428 30250 raft_consensus.cc:3037] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P e86a1f30ca2a45cca27cb5a9c5b57fed [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:31.753428 29982 raft_consensus.cc:3037] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P 1c29740e9a454f18b647ca7f6e6f98f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:31.754652 30250 raft_consensus.cc:2454] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P e86a1f30ca2a45cca27cb5a9c5b57fed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f726dd76bc4159b49e063bad3d9e1b in term 1.
I20260926 12:37:31.754653 29982 raft_consensus.cc:2454] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P 1c29740e9a454f18b647ca7f6e6f98f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5f726dd76bc4159b49e063bad3d9e1b in term 1.
I20260926 12:37:31.754947 30051 leader_election.cc:302] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5f726dd76bc4159b49e063bad3d9e1b, e86a1f30ca2a45cca27cb5a9c5b57fed; no voters:
I20260926 12:37:31.755177 30319 raft_consensus.cc:2781] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:31.755362 30319 raft_consensus.cc:686] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b [term 1 LEADER]: Becoming Leader. State: Replica: a5f726dd76bc4159b49e063bad3d9e1b, State: Running, Role: LEADER
I20260926 12:37:31.755676 30319 consensus_queue.cc:237] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86a1f30ca2a45cca27cb5a9c5b57fed" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36527 } } peers { permanent_uuid: "1c29740e9a454f18b647ca7f6e6f98f2" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 39677 } } peers { permanent_uuid: "a5f726dd76bc4159b49e063bad3d9e1b" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37397 } }
I20260926 12:37:31.758903 29837 catalog_manager.cc:5495] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to a5f726dd76bc4159b49e063bad3d9e1b (127.10.98.2). New cstate: current_term: 1 leader_uuid: "a5f726dd76bc4159b49e063bad3d9e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e86a1f30ca2a45cca27cb5a9c5b57fed" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c29740e9a454f18b647ca7f6e6f98f2" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 39677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5f726dd76bc4159b49e063bad3d9e1b" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 37397 } health_report { overall_health: HEALTHY } } }
I20260926 12:37:31.779469 30312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:31.776292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.10.98.2:37397)
0926 12:37:31.776403 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:31.776406 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:31.776419 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:37:31.776531 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:31.776533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:31.776595 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:31.776688 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:37:31.776893 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.777449 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.778209 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:31.778410 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:31.778482 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:31.778719 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:37:31.779155 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:37:31.779243 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:31.779294 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 12:37:31.779342 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 12:37:31.785377 30250 raft_consensus.cc:1261] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P e86a1f30ca2a45cca27cb5a9c5b57fed [term 1 FOLLOWER]: Refusing update from remote peer a5f726dd76bc4159b49e063bad3d9e1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:37:31.785377 29982 raft_consensus.cc:1261] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P 1c29740e9a454f18b647ca7f6e6f98f2 [term 1 FOLLOWER]: Refusing update from remote peer a5f726dd76bc4159b49e063bad3d9e1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:37:31.786062 30323 consensus_queue.cc:1035] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c29740e9a454f18b647ca7f6e6f98f2" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 39677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:37:31.786239 30319 consensus_queue.cc:1035] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e86a1f30ca2a45cca27cb5a9c5b57fed" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 36527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:37:31.794241 30328 mvcc.cc:205] Tried to move back new op lower bound from 7333585927305736192 to 7333585927193763840. Current Snapshot: MvccSnapshot[applied={T|T < 7333585927305736192}]
I20260926 12:37:31.794593 30327 mvcc.cc:205] Tried to move back new op lower bound from 7333585927305736192 to 7333585927193763840. Current Snapshot: MvccSnapshot[applied={T|T < 7333585927305736192}]
I20260926 12:37:31.801518 29837 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:37:31.801709 29837 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56408:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:37:31.804102 29837 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 73b769a9896c4e7bb413b4af6c9556d4: Sending DeleteTablet for 3 replicas of tablet 43e78ea4a6964b85b9ba4fa1e15ebfe4
I20260926 12:37:31.804844 30230 tablet_service.cc:1514] Processing DeleteTablet for tablet 43e78ea4a6964b85b9ba4fa1e15ebfe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:37:31 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:49490
I20260926 12:37:31.804870 29962 tablet_service.cc:1514] Processing DeleteTablet for tablet 43e78ea4a6964b85b9ba4fa1e15ebfe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:37:31 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:42458
I20260926 12:37:31.805397 30335 tablet_replica.cc:330] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P 1c29740e9a454f18b647ca7f6e6f98f2: stopping tablet replica
I20260926 12:37:31.805546 30096 tablet_service.cc:1514] Processing DeleteTablet for tablet 43e78ea4a6964b85b9ba4fa1e15ebfe4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:37:31 UTC) from {username='kudu', principal='kudu/127.10.98.62@KRBTEST.COM'} at 127.0.0.1:36928
I20260926 12:37:31.805686 30336 tablet_replica.cc:330] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P e86a1f30ca2a45cca27cb5a9c5b57fed: stopping tablet replica
I20260926 12:37:31.805830 30335 raft_consensus.cc:2229] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P 1c29740e9a454f18b647ca7f6e6f98f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:37:31.805939 30336 raft_consensus.cc:2229] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P e86a1f30ca2a45cca27cb5a9c5b57fed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:37:31.806221 30336 raft_consensus.cc:2258] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P e86a1f30ca2a45cca27cb5a9c5b57fed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:37:31.806221 30335 raft_consensus.cc:2258] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P 1c29740e9a454f18b647ca7f6e6f98f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:37:31.806283 30337 tablet_replica.cc:330] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b: stopping tablet replica
I20260926 12:37:31.806602 30337 raft_consensus.cc:2229] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:37:31.806670 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 29899
I20260926 12:37:31.808463 30336 ts_tablet_manager.cc:1905] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P e86a1f30ca2a45cca27cb5a9c5b57fed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:37:31.808583 30337 raft_consensus.cc:2258] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:37:31.809818 30337 ts_tablet_manager.cc:1905] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:37:31.811416 30336 ts_tablet_manager.cc:1918] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P e86a1f30ca2a45cca27cb5a9c5b57fed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:37:31.811563 30336 log.cc:1196] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P e86a1f30ca2a45cca27cb5a9c5b57fed: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/43e78ea4a6964b85b9ba4fa1e15ebfe4
I20260926 12:37:31.811914 30336 ts_tablet_manager.cc:1939] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P e86a1f30ca2a45cca27cb5a9c5b57fed: Deleting consensus metadata
I20260926 12:37:31.812532 30337 ts_tablet_manager.cc:1918] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:37:31.812620 30337 log.cc:1196] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/43e78ea4a6964b85b9ba4fa1e15ebfe4
I20260926 12:37:31.812906 30337 ts_tablet_manager.cc:1939] T 43e78ea4a6964b85b9ba4fa1e15ebfe4 P a5f726dd76bc4159b49e063bad3d9e1b: Deleting consensus metadata
I20260926 12:37:31.813920 29823 catalog_manager.cc:4841] TS e86a1f30ca2a45cca27cb5a9c5b57fed (127.10.98.3:36527): tablet 43e78ea4a6964b85b9ba4fa1e15ebfe4 (table test-table [id=2b23a7bcb1f440afa55ed04ac8715180]) successfully deleted
W20260926 12:37:31.814010 29822 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.98.1:39677 (error 108)
I20260926 12:37:31.814154 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 30033
I20260926 12:37:31.815374 29824 catalog_manager.cc:4841] TS a5f726dd76bc4159b49e063bad3d9e1b (127.10.98.2:37397): tablet 43e78ea4a6964b85b9ba4fa1e15ebfe4 (table test-table [id=2b23a7bcb1f440afa55ed04ac8715180]) successfully deleted
I20260926 12:37:31.815483 30308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:31.815154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:39677 (local address 127.0.0.1:42462)
0926 12:37:31.815279 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:31.815344 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.1:39677: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 12:37:31.815626 29822 catalog_manager.cc:4584] TS 1c29740e9a454f18b647ca7f6e6f98f2 (127.10.98.1:39677): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 43e78ea4a6964b85b9ba4fa1e15ebfe4: Network error: Client connection negotiation failed: client connection to 127.10.98.1:39677: connect: Connection refused (error 111)
I20260926 12:37:31.819795 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 30167
I20260926 12:37:31.824733 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 29805
2026-09-26T12:37:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2231 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T12:37:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:31Z Disabled control of system clock
I20260926 12:37:31.852744 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:35729
--webserver_interface=127.10.98.62
--webserver_port=0
--builtin_ntp_servers=127.10.98.20:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:35729
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 12:37:31.960388 30342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:31.960670 30342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:31.960705 30342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:31.964680 30342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:31.964740 30342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:31.964756 30342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:31.964771 30342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:31.964783 30342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:31.969434 30342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:35729
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:35729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:31.970443 30342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:31.971807 30342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:31.979950 30347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:31.980053 30342 server_base.cc:1034] running on GCE node
W20260926 12:37:31.979950 30350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:31.979985 30348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:31.980789 30342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:31.981904 30342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:31.983085 30342 hybrid_clock.cc:648] HybridClock initialized: now 1790426251983069 us; error 28 us; skew 500 ppm
I20260926 12:37:31.984812 30342 webserver.cc:460] Webserver started at http://127.10.98.62:44239/ using document root <none> and password file <none>
I20260926 12:37:31.985409 30342 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:31.985481 30342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:31.985685 30342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:31.987680 30342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "bb4aad98433549a2a643f1743e59120b"
format_stamp: "Formatted at 2026-09-26 12:37:31 on dist-test-slave-t7tk"
server_key: "93e7636478d113bc395acb4ec5f8df3b"
server_key_iv: "2d33637de86812314f5cae1b4d89acad"
server_key_version: "encryptionkey@0"
I20260926 12:37:31.988209 30342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "bb4aad98433549a2a643f1743e59120b"
format_stamp: "Formatted at 2026-09-26 12:37:31 on dist-test-slave-t7tk"
server_key: "93e7636478d113bc395acb4ec5f8df3b"
server_key_iv: "2d33637de86812314f5cae1b4d89acad"
server_key_version: "encryptionkey@0"
I20260926 12:37:31.991725 30342 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 12:37:31.994172 30356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:31.995425 30342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:37:31.995584 30342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "bb4aad98433549a2a643f1743e59120b"
format_stamp: "Formatted at 2026-09-26 12:37:31 on dist-test-slave-t7tk"
server_key: "93e7636478d113bc395acb4ec5f8df3b"
server_key_iv: "2d33637de86812314f5cae1b4d89acad"
server_key_version: "encryptionkey@0"
I20260926 12:37:31.995687 30342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:32.004856 30342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:32.005594 30342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:32.005762 30342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:32.012575 30342 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:35729
I20260926 12:37:32.012616 30408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:35729 every 8 connection(s)
I20260926 12:37:32.013610 30342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:32.016769 30409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:32.018069 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 30342
I20260926 12:37:32.018205 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:32.018528 10632 external_mini_cluster.cc:1428] Setting key b9cd494e52fb39961370e164efd2f511
I20260926 12:37:32.023406 30409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b: Bootstrap starting.
I20260926 12:37:32.025740 30409 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:32.026556 30409 log.cc:824] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:32.028667 30409 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b: No bootstrap required, opened a new log
I20260926 12:37:32.028667 30412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:32.020216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46460 (local address 127.10.98.62:35729)
0926 12:37:32.020988 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:32.021005 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:32.021045 (+ 40us) server_negotiation.cc:407] Connection header received
0926 12:37:32.021842 (+ 797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:32.021863 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:32.022289 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:32.022664 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:32.023135 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.024151 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.025162 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.025658 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.025823 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:32.026774 (+ 951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:32.026780 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:32.026793 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:32.027292 (+ 499us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:32.027367 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:32.027389 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:32.027476 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:32.027615 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 12:37:32.027884 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":146,"threads_started":1}
I20260926 12:37:32.032472 30409 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb4aad98433549a2a643f1743e59120b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35729 } }
I20260926 12:37:32.032727 30409 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:32.032783 30409 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb4aad98433549a2a643f1743e59120b, State: Initialized, Role: FOLLOWER
I20260926 12:37:32.033228 30409 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb4aad98433549a2a643f1743e59120b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35729 } }
I20260926 12:37:32.033368 30409 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:32.033433 30409 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:32.033543 30409 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:32.034466 30409 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb4aad98433549a2a643f1743e59120b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35729 } }
I20260926 12:37:32.034812 30409 leader_election.cc:302] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb4aad98433549a2a643f1743e59120b; no voters:
I20260926 12:37:32.035161 30409 leader_election.cc:288] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:32.035265 30414 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:32.035511 30414 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b [term 1 LEADER]: Becoming Leader. State: Replica: bb4aad98433549a2a643f1743e59120b, State: Running, Role: LEADER
I20260926 12:37:32.035789 30414 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb4aad98433549a2a643f1743e59120b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35729 } }
I20260926 12:37:32.036312 30409 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:32.037338 30415 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb4aad98433549a2a643f1743e59120b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb4aad98433549a2a643f1743e59120b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35729 } } }
I20260926 12:37:32.037464 30415 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:32.037727 30416 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb4aad98433549a2a643f1743e59120b. Latest consensus state: current_term: 1 leader_uuid: "bb4aad98433549a2a643f1743e59120b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb4aad98433549a2a643f1743e59120b" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35729 } } }
I20260926 12:37:32.037818 30416 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:32.037838 30423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:32.040493 30423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:32.045861 30423 catalog_manager.cc:1347] Generated new cluster ID: a69aa1e3fe5f4e9e988dc3b45378f755
I20260926 12:37:32.045931 30423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:32.069376 30423 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:32.070343 30423 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:32.079041 30423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b: Generated new TSK 0
I20260926 12:37:32.079886 30423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:37:32.085440 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.1:0
--local_ip_for_outbound_sockets=127.10.98.1
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:35729
--builtin_ntp_servers=127.10.98.20:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 12:37:32.199930 30433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:32.200214 30433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:32.200268 30433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:32.204371 30433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:32.204435 30433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:32.204509 30433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.1
I20260926 12:37:32.209326 30433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.10.98.1
--webserver_port=0
--tserver_master_addrs=127.10.98.62:35729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.1
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:32.210420 30433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:32.211753 30433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:32.221364 30438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:32.221381 30439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:32.221618 30433 server_base.cc:1034] running on GCE node
W20260926 12:37:32.221395 30441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:32.222137 30433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:32.222770 30433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:32.223945 30433 hybrid_clock.cc:648] HybridClock initialized: now 1790426252223893 us; error 50 us; skew 500 ppm
I20260926 12:37:32.225759 30433 webserver.cc:460] Webserver started at http://127.10.98.1:44779/ using document root <none> and password file <none>
I20260926 12:37:32.226311 30433 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:32.226377 30433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:32.226562 30433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:32.228595 30433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-0/data/instance:
uuid: "7de35f6bf3d643b3932b623b34e4945b"
format_stamp: "Formatted at 2026-09-26 12:37:32 on dist-test-slave-t7tk"
server_key: "4abb2f4006668fde22c4e0ff361c623a"
server_key_iv: "6a60aa4d8bf5342623120ef453c34370"
server_key_version: "encryptionkey@0"
I20260926 12:37:32.229171 30433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance:
uuid: "7de35f6bf3d643b3932b623b34e4945b"
format_stamp: "Formatted at 2026-09-26 12:37:32 on dist-test-slave-t7tk"
server_key: "4abb2f4006668fde22c4e0ff361c623a"
server_key_iv: "6a60aa4d8bf5342623120ef453c34370"
server_key_version: "encryptionkey@0"
I20260926 12:37:32.232762 30433 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260926 12:37:32.235229 30447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:32.236450 30433 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:37:32.236568 30433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-0/wal
uuid: "7de35f6bf3d643b3932b623b34e4945b"
format_stamp: "Formatted at 2026-09-26 12:37:32 on dist-test-slave-t7tk"
server_key: "4abb2f4006668fde22c4e0ff361c623a"
server_key_iv: "6a60aa4d8bf5342623120ef453c34370"
server_key_version: "encryptionkey@0"
I20260926 12:37:32.236677 30433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:32.250901 30433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:32.251572 30433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:32.251735 30433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:32.252292 30433 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:37:32.253228 30433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:37:32.253276 30433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:32.253331 30433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:37:32.253357 30433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:32.261876 30433 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.1:34649
I20260926 12:37:32.261898 30560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.1:34649 every 8 connection(s)
I20260926 12:37:32.262734 30433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-0/data/info.pb
I20260926 12:37:32.271492 30412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:32.264468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.1:46417 (local address 127.10.98.62:35729)
0926 12:37:32.264633 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:32.264637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:32.265697 (+ 1060us) server_negotiation.cc:407] Connection header received
0926 12:37:32.267042 (+ 1345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:32.267045 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:32.267093 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:32.267172 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:32.268063 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.268633 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.269818 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.270052 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.270141 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:32.271120 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:32.271123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:32.271125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:32.271138 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:32.271157 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:32.271165 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:32.271224 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:32.271326 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 12:37:32.271378 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:37:32.272344 30563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:32.264858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:35729 (local address 127.10.98.1:46417)
0926 12:37:32.265502 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:32.265583 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:32.266807 (+ 1224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:32.267323 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:32.267330 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:32.267415 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:32.267934 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:32.267946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.268757 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.268760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:32.269696 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:32.269705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.270178 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.270180 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:32.270319 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:32.270885 (+ 566us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:37:32.270886 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:32.270925 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:37:32.270986 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:32.271229 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:32.271234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:32.271237 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 12:37:32.271320 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 12:37:32.271576 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":97,"threads_started":1}
I20260926 12:37:32.272410 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 30433
I20260926 12:37:32.272524 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-0/wal/instance
I20260926 12:37:32.272799 10632 external_mini_cluster.cc:1428] Setting key 6091056a2c4ca5f408eecad51c364810
I20260926 12:37:32.273545 30561 heartbeater.cc:347] Connected to a master server at 127.10.98.62:35729
I20260926 12:37:32.273779 30561 heartbeater.cc:464] Registering TS with master...
I20260926 12:37:32.274298 30561 heartbeater.cc:511] Master 127.10.98.62:35729 requested a full tablet report, sending...
I20260926 12:37:32.274619 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.2:0
--local_ip_for_outbound_sockets=127.10.98.2
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:35729
--builtin_ntp_servers=127.10.98.20:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 12:37:32.275848 30373 ts_manager.cc:194] Registered new tserver with Master: 7de35f6bf3d643b3932b623b34e4945b (127.10.98.1:34649)
I20260926 12:37:32.276975 30373 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.98.1:46417
W20260926 12:37:32.384883 30564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:32.385197 30564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:32.385249 30564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:32.389251 30564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:32.389305 30564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:32.389359 30564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.2
I20260926 12:37:32.394203 30564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.10.98.2
--webserver_port=0
--tserver_master_addrs=127.10.98.62:35729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.2
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:32.395319 30564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:32.396709 30564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:32.406433 30569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:32.406522 30564 server_base.cc:1034] running on GCE node
W20260926 12:37:32.406567 30572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:32.406410 30570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:32.407094 30564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:32.407766 30564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:32.408928 30564 hybrid_clock.cc:648] HybridClock initialized: now 1790426252408909 us; error 34 us; skew 500 ppm
I20260926 12:37:32.410794 30564 webserver.cc:460] Webserver started at http://127.10.98.2:42049/ using document root <none> and password file <none>
I20260926 12:37:32.411387 30564 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:32.411456 30564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:32.411646 30564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:32.413843 30564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-1/data/instance:
uuid: "657f255668fe48a3a66c5340d33a0df9"
format_stamp: "Formatted at 2026-09-26 12:37:32 on dist-test-slave-t7tk"
server_key: "162eeb0a530efd335c64b6ca545eff24"
server_key_iv: "010ed0befdde74572ae603120e49c02a"
server_key_version: "encryptionkey@0"
I20260926 12:37:32.414397 30564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance:
uuid: "657f255668fe48a3a66c5340d33a0df9"
format_stamp: "Formatted at 2026-09-26 12:37:32 on dist-test-slave-t7tk"
server_key: "162eeb0a530efd335c64b6ca545eff24"
server_key_iv: "010ed0befdde74572ae603120e49c02a"
server_key_version: "encryptionkey@0"
I20260926 12:37:32.418104 30564 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:37:32.420573 30578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:32.421952 30564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:37:32.422056 30564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-1/wal
uuid: "657f255668fe48a3a66c5340d33a0df9"
format_stamp: "Formatted at 2026-09-26 12:37:32 on dist-test-slave-t7tk"
server_key: "162eeb0a530efd335c64b6ca545eff24"
server_key_iv: "010ed0befdde74572ae603120e49c02a"
server_key_version: "encryptionkey@0"
I20260926 12:37:32.422160 30564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:32.441416 30564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:32.442143 30564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:32.442318 30564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:32.442910 30564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:37:32.443862 30564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:37:32.443909 30564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:32.443953 30564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:37:32.443981 30564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:32.452138 30564 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.2:40751
I20260926 12:37:32.452194 30691 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.2:40751 every 8 connection(s)
I20260926 12:37:32.453043 30564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-1/data/info.pb
I20260926 12:37:32.461875 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 30564
I20260926 12:37:32.462005 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-1/wal/instance
I20260926 12:37:32.461990 30412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:32.454830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.2:54291 (local address 127.10.98.62:35729)
0926 12:37:32.454997 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:32.455001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:32.456044 (+ 1043us) server_negotiation.cc:407] Connection header received
0926 12:37:32.457463 (+ 1419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:32.457465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:32.457510 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:32.457588 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:32.458461 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.458978 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.460098 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.460372 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.460460 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:32.461497 (+ 1037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:32.461500 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:32.461501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:32.461514 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:32.461534 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:32.461552 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:32.461608 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:32.461777 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 12:37:32.461841 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 12:37:32.462316 10632 external_mini_cluster.cc:1428] Setting key 3c04c1207924d719764e9ce07e74d50e
I20260926 12:37:32.462860 30694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:32.455236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:35729 (local address 127.10.98.2:54291)
0926 12:37:32.455887 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:32.455928 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:32.457287 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:32.457743 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:32.457750 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:32.457834 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:32.458356 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:32.458369 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.459085 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.459087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:32.459980 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:32.459990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.460515 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.460518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:32.460673 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:32.461253 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:37:32.461254 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:32.461295 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:37:32.461360 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:32.461617 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:32.461623 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:32.461627 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:37:32.461746 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 12:37:32.462026 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":104,"threads_started":1}
I20260926 12:37:32.464128 30692 heartbeater.cc:347] Connected to a master server at 127.10.98.62:35729
I20260926 12:37:32.464308 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.10.98.3:0
--local_ip_for_outbound_sockets=127.10.98.3
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:35729
--builtin_ntp_servers=127.10.98.20:33941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 12:37:32.464349 30692 heartbeater.cc:464] Registering TS with master...
I20260926 12:37:32.464859 30692 heartbeater.cc:511] Master 127.10.98.62:35729 requested a full tablet report, sending...
I20260926 12:37:32.465970 30373 ts_manager.cc:194] Registered new tserver with Master: 657f255668fe48a3a66c5340d33a0df9 (127.10.98.2:40751)
I20260926 12:37:32.466521 30373 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.98.2:54291
W20260926 12:37:32.574251 30695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:32.574536 30695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:32.574574 30695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:32.578550 30695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:32.578603 30695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:32.578655 30695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.10.98.3
I20260926 12:37:32.583580 30695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:33941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.10.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.10.98.3
--webserver_port=0
--tserver_master_addrs=127.10.98.62:35729
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.10.98.3
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:32.584635 30695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:32.586012 30695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:32.595275 30700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:32.595299 30701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:32.595358 30695 server_base.cc:1034] running on GCE node
W20260926 12:37:32.595595 30703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:32.595970 30695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:32.596619 30695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:32.597774 30695 hybrid_clock.cc:648] HybridClock initialized: now 1790426252597757 us; error 30 us; skew 500 ppm
I20260926 12:37:32.599545 30695 webserver.cc:460] Webserver started at http://127.10.98.3:33543/ using document root <none> and password file <none>
I20260926 12:37:32.600128 30695 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:32.600224 30695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:32.600512 30695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:32.602617 30695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-2/data/instance:
uuid: "1297310e48ff46ef8db8929eae908d2a"
format_stamp: "Formatted at 2026-09-26 12:37:32 on dist-test-slave-t7tk"
server_key: "120ab40cda715771f0e63460e9b0b6c0"
server_key_iv: "bbcc3b1f45bea916ba888efc83f26a61"
server_key_version: "encryptionkey@0"
I20260926 12:37:32.603159 30695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance:
uuid: "1297310e48ff46ef8db8929eae908d2a"
format_stamp: "Formatted at 2026-09-26 12:37:32 on dist-test-slave-t7tk"
server_key: "120ab40cda715771f0e63460e9b0b6c0"
server_key_iv: "bbcc3b1f45bea916ba888efc83f26a61"
server_key_version: "encryptionkey@0"
I20260926 12:37:32.606742 30695 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:37:32.609180 30709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:32.610399 30695 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:37:32.610513 30695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-2/wal
uuid: "1297310e48ff46ef8db8929eae908d2a"
format_stamp: "Formatted at 2026-09-26 12:37:32 on dist-test-slave-t7tk"
server_key: "120ab40cda715771f0e63460e9b0b6c0"
server_key_iv: "bbcc3b1f45bea916ba888efc83f26a61"
server_key_version: "encryptionkey@0"
I20260926 12:37:32.610610 30695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:32.642627 30695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:32.643365 30695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:32.643539 30695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:32.644145 30695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 12:37:32.645114 30695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 12:37:32.645161 30695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:32.645212 30695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 12:37:32.645241 30695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:32.653723 30695 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.3:33223
I20260926 12:37:32.653751 30822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.3:33223 every 8 connection(s)
I20260926 12:37:32.654649 30695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-2/data/info.pb
I20260926 12:37:32.660701 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 30695
I20260926 12:37:32.660822 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-2/wal/instance
I20260926 12:37:32.661070 10632 external_mini_cluster.cc:1428] Setting key 38209e26f05b7d5bdacc1e4ac39a9cea
I20260926 12:37:32.663802 30412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:32.656471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:56239 (local address 127.10.98.62:35729)
0926 12:37:32.656610 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:32.656614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:32.657718 (+ 1104us) server_negotiation.cc:407] Connection header received
0926 12:37:32.659043 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:32.659046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:32.659094 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:32.659166 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:32.659975 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.660488 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.661931 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.662149 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.662237 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:32.663396 (+ 1159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:32.663399 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:32.663401 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:32.663413 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:32.663432 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:32.663440 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:32.663494 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:32.663645 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 12:37:32.663693 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 12:37:32.664769 30825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:32.656866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:35729 (local address 127.10.98.3:56239)
0926 12:37:32.657588 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:32.657632 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:32.658857 (+ 1225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:32.659300 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:32.659306 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:32.659391 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:32.659871 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:32.659883 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.660695 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.660698 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:32.661785 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:32.661796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.662391 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.662394 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:32.662543 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:32.663161 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 12:37:32.663163 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:32.663202 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 12:37:32.663266 (+ 64us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:32.663501 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:32.663505 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:32.663509 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 12:37:32.663641 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 12:37:32.663861 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":105,"threads_started":1}
I20260926 12:37:32.665804 30823 heartbeater.cc:347] Connected to a master server at 127.10.98.62:35729
I20260926 12:37:32.666019 30823 heartbeater.cc:464] Registering TS with master...
I20260926 12:37:32.666514 30823 heartbeater.cc:511] Master 127.10.98.62:35729 requested a full tablet report, sending...
I20260926 12:37:32.667558 30373 ts_manager.cc:194] Registered new tserver with Master: 1297310e48ff46ef8db8929eae908d2a (127.10.98.3:33223)
I20260926 12:37:32.668121 30373 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.10.98.3:56239
I20260926 12:37:32.674672 10632 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 12:37:32.680081 30412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:32.676564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.10.98.62:35729)
0926 12:37:32.676678 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:32.676682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:32.676849 (+ 167us) server_negotiation.cc:407] Connection header received
0926 12:37:32.676945 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:32.676947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:32.676986 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:32.677058 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:32.677331 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.677837 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.678649 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.678853 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.678927 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:32.679657 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:32.679660 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:32.679662 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:32.679673 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:32.679692 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:32.679700 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:32.679753 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:32.679910 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 12:37:32.679962 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 12:37:32.682813 30373 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 12:37:32.685851 30373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 12:37:32.699836 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:32.695076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.10.98.2:40751)
0926 12:37:32.695749 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:32.695762 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:32.695782 (+ 20us) server_negotiation.cc:407] Connection header received
0926 12:37:32.695848 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:32.695853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:32.696040 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:32.696177 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:32.696383 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.697435 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.698613 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.699203 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.699296 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:32.699507 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:32.699596 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 12:37:32.699689 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":115,"threads_started":1}
I20260926 12:37:32.699836 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:32.694675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:40751 (local address 127.0.0.1:36126)
0926 12:37:32.695442 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:32.695455 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:32.695529 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:32.696169 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:32.696172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:32.696190 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:32.696276 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:32.696282 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.697573 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.697576 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:32.698509 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:32.698520 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.699308 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.699312 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:32.699414 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:32.699450 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 12:37:32.699544 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 12:37:32.699663 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"mutex_wait_us":21,"thread_start_us":105,"threads_started":1}
I20260926 12:37:32.701308 30834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:32.693982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:34649 (local address 127.0.0.1:42896)
0926 12:37:32.694298 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:32.694343 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:32.694477 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:32.695317 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:32.695323 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:32.695349 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:32.695633 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:32.695640 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.697187 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.697191 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:32.698503 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:32.698514 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.700873 (+ 2359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.700877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:32.700985 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:32.701010 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 12:37:32.701074 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 12:37:32.701170 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":67,"spinlock_wait_cycles":1920,"thread_start_us":74,"threads_started":1}
I20260926 12:37:32.701370 30835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:32.694101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42896 (local address 127.10.98.1:34649)
0926 12:37:32.694787 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:32.694800 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:32.694826 (+ 26us) server_negotiation.cc:407] Connection header received
0926 12:37:32.694905 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:32.694911 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:32.695134 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:32.695780 (+ 646us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:32.695928 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.697074 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.700012 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.700779 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.700878 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:32.701059 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:32.701129 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 12:37:32.701221 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":118,"threads_started":1}
I20260926 12:37:32.702845 30836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:32.694177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.3:33223 (local address 127.0.0.1:33132)
0926 12:37:32.695275 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:32.695288 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:32.695396 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:32.695788 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:32.695790 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:32.695807 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:32.695907 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:32.695914 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.696953 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.696956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:32.698504 (+ 1548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:32.698514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.702406 (+ 3892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.702410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:32.702548 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:32.702573 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 12:37:32.702651 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 12:37:32.702718 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"mutex_wait_us":45,"spinlock_wait_cycles":768,"thread_start_us":89,"threads_started":1}
I20260926 12:37:32.702937 30838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:32.694766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33132 (local address 127.10.98.3:33223)
0926 12:37:32.695252 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:32.695263 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:32.695377 (+ 114us) server_negotiation.cc:407] Connection header received
0926 12:37:32.695525 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:32.695529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:32.695667 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:32.695796 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:32.696012 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.696844 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.701544 (+ 4700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.702280 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.702400 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:32.702557 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:32.702687 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 12:37:32.702778 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":92,"threads_started":1}
I20260926 12:37:32.703263 30626 tablet_service.cc:1467] Processing CreateTablet for tablet 1c41411426554d7b8703b69852f870ab (DEFAULT_TABLE table=test-table [id=ed48e921f0934b818745631bd2cfc210]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:37:32.704232 30626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c41411426554d7b8703b69852f870ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:32.703905 30495 tablet_service.cc:1467] Processing CreateTablet for tablet 1c41411426554d7b8703b69852f870ab (DEFAULT_TABLE table=test-table [id=ed48e921f0934b818745631bd2cfc210]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:37:32.704802 30495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c41411426554d7b8703b69852f870ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:32.710199 30840 tablet_bootstrap.cc:492] T 1c41411426554d7b8703b69852f870ab P 657f255668fe48a3a66c5340d33a0df9: Bootstrap starting.
I20260926 12:37:32.711006 30841 tablet_bootstrap.cc:492] T 1c41411426554d7b8703b69852f870ab P 7de35f6bf3d643b3932b623b34e4945b: Bootstrap starting.
I20260926 12:37:32.711694 30757 tablet_service.cc:1467] Processing CreateTablet for tablet 1c41411426554d7b8703b69852f870ab (DEFAULT_TABLE table=test-table [id=ed48e921f0934b818745631bd2cfc210]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 12:37:32.712430 30840 tablet_bootstrap.cc:654] T 1c41411426554d7b8703b69852f870ab P 657f255668fe48a3a66c5340d33a0df9: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:32.712482 30757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c41411426554d7b8703b69852f870ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:32.713034 30841 tablet_bootstrap.cc:654] T 1c41411426554d7b8703b69852f870ab P 7de35f6bf3d643b3932b623b34e4945b: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:32.713340 30840 log.cc:824] T 1c41411426554d7b8703b69852f870ab P 657f255668fe48a3a66c5340d33a0df9: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:32.713828 30841 log.cc:824] T 1c41411426554d7b8703b69852f870ab P 7de35f6bf3d643b3932b623b34e4945b: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:32.715473 30840 tablet_bootstrap.cc:492] T 1c41411426554d7b8703b69852f870ab P 657f255668fe48a3a66c5340d33a0df9: No bootstrap required, opened a new log
I20260926 12:37:32.715694 30841 tablet_bootstrap.cc:492] T 1c41411426554d7b8703b69852f870ab P 7de35f6bf3d643b3932b623b34e4945b: No bootstrap required, opened a new log
I20260926 12:37:32.715785 30840 ts_tablet_manager.cc:1397] T 1c41411426554d7b8703b69852f870ab P 657f255668fe48a3a66c5340d33a0df9: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260926 12:37:32.715863 30841 ts_tablet_manager.cc:1397] T 1c41411426554d7b8703b69852f870ab P 7de35f6bf3d643b3932b623b34e4945b: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260926 12:37:32.718923 30841 raft_consensus.cc:348] T 1c41411426554d7b8703b69852f870ab P 7de35f6bf3d643b3932b623b34e4945b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de35f6bf3d643b3932b623b34e4945b" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34649 } } peers { permanent_uuid: "657f255668fe48a3a66c5340d33a0df9" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40751 } } peers { permanent_uuid: "1297310e48ff46ef8db8929eae908d2a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33223 } }
I20260926 12:37:32.719133 30841 raft_consensus.cc:374] T 1c41411426554d7b8703b69852f870ab P 7de35f6bf3d643b3932b623b34e4945b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:32.719193 30841 raft_consensus.cc:729] T 1c41411426554d7b8703b69852f870ab P 7de35f6bf3d643b3932b623b34e4945b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7de35f6bf3d643b3932b623b34e4945b, State: Initialized, Role: FOLLOWER
I20260926 12:37:32.719621 30844 tablet_bootstrap.cc:492] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a: Bootstrap starting.
I20260926 12:37:32.719660 30841 consensus_queue.cc:260] T 1c41411426554d7b8703b69852f870ab P 7de35f6bf3d643b3932b623b34e4945b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de35f6bf3d643b3932b623b34e4945b" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34649 } } peers { permanent_uuid: "657f255668fe48a3a66c5340d33a0df9" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40751 } } peers { permanent_uuid: "1297310e48ff46ef8db8929eae908d2a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33223 } }
I20260926 12:37:32.719774 30840 raft_consensus.cc:348] T 1c41411426554d7b8703b69852f870ab P 657f255668fe48a3a66c5340d33a0df9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de35f6bf3d643b3932b623b34e4945b" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34649 } } peers { permanent_uuid: "657f255668fe48a3a66c5340d33a0df9" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40751 } } peers { permanent_uuid: "1297310e48ff46ef8db8929eae908d2a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33223 } }
I20260926 12:37:32.720036 30840 raft_consensus.cc:374] T 1c41411426554d7b8703b69852f870ab P 657f255668fe48a3a66c5340d33a0df9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:32.720094 30840 raft_consensus.cc:729] T 1c41411426554d7b8703b69852f870ab P 657f255668fe48a3a66c5340d33a0df9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 657f255668fe48a3a66c5340d33a0df9, State: Initialized, Role: FOLLOWER
I20260926 12:37:32.720541 30840 consensus_queue.cc:260] T 1c41411426554d7b8703b69852f870ab P 657f255668fe48a3a66c5340d33a0df9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de35f6bf3d643b3932b623b34e4945b" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34649 } } peers { permanent_uuid: "657f255668fe48a3a66c5340d33a0df9" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40751 } } peers { permanent_uuid: "1297310e48ff46ef8db8929eae908d2a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33223 } }
I20260926 12:37:32.720659 30561 heartbeater.cc:503] Master 127.10.98.62:35729 was elected leader, sending a full tablet report...
I20260926 12:37:32.720640 30841 ts_tablet_manager.cc:1428] T 1c41411426554d7b8703b69852f870ab P 7de35f6bf3d643b3932b623b34e4945b: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:37:32.721494 30692 heartbeater.cc:503] Master 127.10.98.62:35729 was elected leader, sending a full tablet report...
I20260926 12:37:32.722464 30844 tablet_bootstrap.cc:654] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:32.721480 30840 ts_tablet_manager.cc:1428] T 1c41411426554d7b8703b69852f870ab P 657f255668fe48a3a66c5340d33a0df9: Time spent starting tablet: real 0.006s user 0.000s sys 0.005s
I20260926 12:37:32.723192 30844 log.cc:824] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:32.725615 30844 tablet_bootstrap.cc:492] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a: No bootstrap required, opened a new log
I20260926 12:37:32.725791 30844 ts_tablet_manager.cc:1397] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 12:37:32.728926 30844 raft_consensus.cc:348] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de35f6bf3d643b3932b623b34e4945b" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34649 } } peers { permanent_uuid: "657f255668fe48a3a66c5340d33a0df9" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40751 } } peers { permanent_uuid: "1297310e48ff46ef8db8929eae908d2a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33223 } }
I20260926 12:37:32.729166 30844 raft_consensus.cc:374] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:32.729221 30844 raft_consensus.cc:729] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1297310e48ff46ef8db8929eae908d2a, State: Initialized, Role: FOLLOWER
I20260926 12:37:32.729604 30844 consensus_queue.cc:260] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de35f6bf3d643b3932b623b34e4945b" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34649 } } peers { permanent_uuid: "657f255668fe48a3a66c5340d33a0df9" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40751 } } peers { permanent_uuid: "1297310e48ff46ef8db8929eae908d2a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33223 } }
I20260926 12:37:32.730177 30823 heartbeater.cc:503] Master 127.10.98.62:35729 was elected leader, sending a full tablet report...
I20260926 12:37:32.730465 30844 ts_tablet_manager.cc:1428] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 12:37:32.743544 30848 raft_consensus.cc:482] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 12:37:32.743721 30848 raft_consensus.cc:504] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de35f6bf3d643b3932b623b34e4945b" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34649 } } peers { permanent_uuid: "657f255668fe48a3a66c5340d33a0df9" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40751 } } peers { permanent_uuid: "1297310e48ff46ef8db8929eae908d2a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33223 } }
I20260926 12:37:32.744786 30848 leader_election.cc:288] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7de35f6bf3d643b3932b623b34e4945b (127.10.98.1:34649), 657f255668fe48a3a66c5340d33a0df9 (127.10.98.2:40751)
I20260926 12:37:32.748767 30825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:32.744970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:34649 (local address 127.10.98.3:47805)
0926 12:37:32.745078 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:32.745092 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:32.745207 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:32.745455 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:32.745458 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:32.745478 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:32.745601 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:32.745610 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.746567 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.746569 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:32.747683 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:32.747694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.748425 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.748427 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:32.748500 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:32.748520 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 12:37:32.748588 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 12:37:32.748644 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 12:37:32.748807 30835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:32.745115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:47805 (local address 127.10.98.1:34649)
0926 12:37:32.745231 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:32.745235 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:32.745250 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:37:32.745301 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:32.745304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:32.745348 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:32.745410 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:32.745723 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.746456 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.747810 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.748319 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.748417 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:32.748478 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:32.748612 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 12:37:32.748685 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 12:37:32.749152 30839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:32.745343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.10.98.3:48533 (local address 127.10.98.2:40751)
0926 12:37:32.745487 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:32.745491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:32.745781 (+ 290us) server_negotiation.cc:407] Connection header received
0926 12:37:32.745861 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:32.745864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:32.745913 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:32.745985 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 12:37:32.746187 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.746788 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.748042 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.748612 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.748707 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:32.748760 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:32.748965 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 12:37:32.749036 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 12:37:32.749415 30515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c41411426554d7b8703b69852f870ab" candidate_uuid: "1297310e48ff46ef8db8929eae908d2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7de35f6bf3d643b3932b623b34e4945b" is_pre_election: true
I20260926 12:37:32.749526 30849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:32.745247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.2:40751 (local address 127.10.98.3:48533)
0926 12:37:32.745700 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:32.745712 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:32.745790 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:32.745993 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:32.745995 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:32.746005 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 12:37:32.746103 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:32.746107 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.746896 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.746899 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:32.747933 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:32.747940 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.748734 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.748737 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:32.748822 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:32.748843 (+ 21us) client_negotiation.cc:769] Sending connection context
0926 12:37:32.749333 (+ 490us) client_negotiation.cc:240] Negotiation successful
0926 12:37:32.749403 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":126,"threads_started":1}
I20260926 12:37:32.749744 30515 raft_consensus.cc:2454] T 1c41411426554d7b8703b69852f870ab P 7de35f6bf3d643b3932b623b34e4945b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1297310e48ff46ef8db8929eae908d2a in term 0.
I20260926 12:37:32.750005 30646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c41411426554d7b8703b69852f870ab" candidate_uuid: "1297310e48ff46ef8db8929eae908d2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "657f255668fe48a3a66c5340d33a0df9" is_pre_election: true
I20260926 12:37:32.750219 30712 leader_election.cc:302] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1297310e48ff46ef8db8929eae908d2a, 7de35f6bf3d643b3932b623b34e4945b; no voters:
I20260926 12:37:32.750291 30646 raft_consensus.cc:2454] T 1c41411426554d7b8703b69852f870ab P 657f255668fe48a3a66c5340d33a0df9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1297310e48ff46ef8db8929eae908d2a in term 0.
I20260926 12:37:32.750499 30848 raft_consensus.cc:2781] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 12:37:32.750592 30848 raft_consensus.cc:482] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 12:37:32.750643 30848 raft_consensus.cc:3037] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:32.751597 30848 raft_consensus.cc:504] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de35f6bf3d643b3932b623b34e4945b" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34649 } } peers { permanent_uuid: "657f255668fe48a3a66c5340d33a0df9" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40751 } } peers { permanent_uuid: "1297310e48ff46ef8db8929eae908d2a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33223 } }
I20260926 12:37:32.751896 30848 leader_election.cc:288] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a [CANDIDATE]: Term 1 election: Requested vote from peers 7de35f6bf3d643b3932b623b34e4945b (127.10.98.1:34649), 657f255668fe48a3a66c5340d33a0df9 (127.10.98.2:40751)
I20260926 12:37:32.752198 30515 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c41411426554d7b8703b69852f870ab" candidate_uuid: "1297310e48ff46ef8db8929eae908d2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7de35f6bf3d643b3932b623b34e4945b"
I20260926 12:37:32.752278 30646 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c41411426554d7b8703b69852f870ab" candidate_uuid: "1297310e48ff46ef8db8929eae908d2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "657f255668fe48a3a66c5340d33a0df9"
I20260926 12:37:32.752314 30515 raft_consensus.cc:3037] T 1c41411426554d7b8703b69852f870ab P 7de35f6bf3d643b3932b623b34e4945b [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:32.752393 30646 raft_consensus.cc:3037] T 1c41411426554d7b8703b69852f870ab P 657f255668fe48a3a66c5340d33a0df9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:32.753362 30515 raft_consensus.cc:2454] T 1c41411426554d7b8703b69852f870ab P 7de35f6bf3d643b3932b623b34e4945b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1297310e48ff46ef8db8929eae908d2a in term 1.
I20260926 12:37:32.753634 30646 raft_consensus.cc:2454] T 1c41411426554d7b8703b69852f870ab P 657f255668fe48a3a66c5340d33a0df9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1297310e48ff46ef8db8929eae908d2a in term 1.
I20260926 12:37:32.753659 30712 leader_election.cc:302] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1297310e48ff46ef8db8929eae908d2a, 7de35f6bf3d643b3932b623b34e4945b; no voters:
I20260926 12:37:32.753814 30848 raft_consensus.cc:2781] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:32.754026 30848 raft_consensus.cc:686] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a [term 1 LEADER]: Becoming Leader. State: Replica: 1297310e48ff46ef8db8929eae908d2a, State: Running, Role: LEADER
I20260926 12:37:32.754366 30848 consensus_queue.cc:237] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de35f6bf3d643b3932b623b34e4945b" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34649 } } peers { permanent_uuid: "657f255668fe48a3a66c5340d33a0df9" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40751 } } peers { permanent_uuid: "1297310e48ff46ef8db8929eae908d2a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33223 } }
I20260926 12:37:32.757328 30373 catalog_manager.cc:5495] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1297310e48ff46ef8db8929eae908d2a (127.10.98.3). New cstate: current_term: 1 leader_uuid: "1297310e48ff46ef8db8929eae908d2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7de35f6bf3d643b3932b623b34e4945b" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "657f255668fe48a3a66c5340d33a0df9" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1297310e48ff46ef8db8929eae908d2a" member_type: VOTER last_known_addr { host: "127.10.98.3" port: 33223 } health_report { overall_health: HEALTHY } } }
W20260926 12:37:32.763820 30562 tablet.cc:2369] T 1c41411426554d7b8703b69852f870ab P 7de35f6bf3d643b3932b623b34e4945b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:37:32.777077 30838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:32.773664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33144 (local address 127.10.98.3:33223)
0926 12:37:32.773780 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:32.773783 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:32.773797 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:37:32.773901 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:32.773903 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:32.773955 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:32.774042 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:37:32.774226 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.774713 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.775579 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:32.775797 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:32.775892 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:32.776227 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:37:32.776723 (+ 496us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:37:32.776816 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:32.776905 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 12:37:32.776954 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:37:32.782872 30646 raft_consensus.cc:1261] T 1c41411426554d7b8703b69852f870ab P 657f255668fe48a3a66c5340d33a0df9 [term 1 FOLLOWER]: Refusing update from remote peer 1297310e48ff46ef8db8929eae908d2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:37:32.782955 30515 raft_consensus.cc:1261] T 1c41411426554d7b8703b69852f870ab P 7de35f6bf3d643b3932b623b34e4945b [term 1 FOLLOWER]: Refusing update from remote peer 1297310e48ff46ef8db8929eae908d2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:37:32.783558 30850 consensus_queue.cc:1035] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "657f255668fe48a3a66c5340d33a0df9" member_type: VOTER last_known_addr { host: "127.10.98.2" port: 40751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:37:32.783752 30848 consensus_queue.cc:1035] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7de35f6bf3d643b3932b623b34e4945b" member_type: VOTER last_known_addr { host: "127.10.98.1" port: 34649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:37:32.791607 30855 mvcc.cc:205] Tried to move back new op lower bound from 7333585931391860736 to 7333585931284525056. Current Snapshot: MvccSnapshot[applied={T|T < 7333585931391860736}]
I20260926 12:37:32.791923 30854 mvcc.cc:205] Tried to move back new op lower bound from 7333585931391860736 to 7333585931284525056. Current Snapshot: MvccSnapshot[applied={T|T < 7333585931391860736}]
I20260926 12:37:32.799813 30371 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:37:32.800022 30371 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 12:37:32.802616 30371 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb4aad98433549a2a643f1743e59120b: Sending DeleteTablet for 3 replicas of tablet 1c41411426554d7b8703b69852f870ab
I20260926 12:37:32.803447 30626 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c41411426554d7b8703b69852f870ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:37:32 UTC) from {username='slave'} at 127.0.0.1:36126
I20260926 12:37:32.803577 30495 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c41411426554d7b8703b69852f870ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:37:32 UTC) from {username='slave'} at 127.0.0.1:42896
I20260926 12:37:32.803979 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 30433
I20260926 12:37:32.803978 30757 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c41411426554d7b8703b69852f870ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 12:37:32 UTC) from {username='slave'} at 127.0.0.1:33132
I20260926 12:37:32.804584 30863 tablet_replica.cc:330] T 1c41411426554d7b8703b69852f870ab P 657f255668fe48a3a66c5340d33a0df9: stopping tablet replica
I20260926 12:37:32.804924 30863 raft_consensus.cc:2229] T 1c41411426554d7b8703b69852f870ab P 657f255668fe48a3a66c5340d33a0df9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 12:37:32.805222 30865 tablet_replica.cc:330] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a: stopping tablet replica
I20260926 12:37:32.805291 30863 raft_consensus.cc:2258] T 1c41411426554d7b8703b69852f870ab P 657f255668fe48a3a66c5340d33a0df9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:37:32.806066 30865 raft_consensus.cc:2229] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a [term 1 LEADER]: Raft consensus shutting down.
I20260926 12:37:32.806550 30865 raft_consensus.cc:2258] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 12:37:32.806833 30863 ts_tablet_manager.cc:1905] T 1c41411426554d7b8703b69852f870ab P 657f255668fe48a3a66c5340d33a0df9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:37:32.808014 30865 ts_tablet_manager.cc:1905] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 12:37:32.810062 30863 ts_tablet_manager.cc:1918] T 1c41411426554d7b8703b69852f870ab P 657f255668fe48a3a66c5340d33a0df9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:37:32.810164 30863 log.cc:1196] T 1c41411426554d7b8703b69852f870ab P 657f255668fe48a3a66c5340d33a0df9: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-1/wal/wals/1c41411426554d7b8703b69852f870ab
I20260926 12:37:32.810446 30863 ts_tablet_manager.cc:1939] T 1c41411426554d7b8703b69852f870ab P 657f255668fe48a3a66c5340d33a0df9: Deleting consensus metadata
I20260926 12:37:32.810781 30865 ts_tablet_manager.cc:1918] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 12:37:32.810940 30865 log.cc:1196] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a: Deleting WAL directory at /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790425997379116-10632-0/minicluster-data/ts-2/wal/wals/1c41411426554d7b8703b69852f870ab
I20260926 12:37:32.811271 30865 ts_tablet_manager.cc:1939] T 1c41411426554d7b8703b69852f870ab P 1297310e48ff46ef8db8929eae908d2a: Deleting consensus metadata
I20260926 12:37:32.812347 30360 catalog_manager.cc:4841] TS 1297310e48ff46ef8db8929eae908d2a (127.10.98.3:33223): tablet 1c41411426554d7b8703b69852f870ab (table test-table [id=ed48e921f0934b818745631bd2cfc210]) successfully deleted
W20260926 12:37:32.813110 30359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.10.98.1:34649 (error 108)
I20260926 12:37:32.813169 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 30564
I20260926 12:37:32.813567 30359 catalog_manager.cc:4841] TS 657f255668fe48a3a66c5340d33a0df9 (127.10.98.2:40751): tablet 1c41411426554d7b8703b69852f870ab (table test-table [id=ed48e921f0934b818745631bd2cfc210]) successfully deleted
I20260926 12:37:32.814682 30836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:32.814381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.1:34649 (local address 127.0.0.1:42898)
0926 12:37:32.814521 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:32.814594 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.1:34649: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 12:37:32.814851 30359 catalog_manager.cc:4584] TS 7de35f6bf3d643b3932b623b34e4945b (127.10.98.1:34649): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c41411426554d7b8703b69852f870ab: Network error: Client connection negotiation failed: client connection to 127.10.98.1:34649: connect: Connection refused (error 111)
I20260926 12:37:32.821026 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 30695
I20260926 12:37:32.825909 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 30342
2026-09-26T12:37:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1000 ms)
[----------] 6 tests from EncryptionPolicyTest (10169 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:32 dist-test-slave-t7tk krb5kdc[30869](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:32 dist-test-slave-t7tk krb5kdc[30869](info): set up 2 sockets
Sep 26 12:37:32 dist-test-slave-t7tk krb5kdc[30869](info): commencing operation
krb5kdc: starting...
W20260926 12:37:33.794572 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:33 dist-test-slave-t7tk krb5kdc[30869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:37:33.912360 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42703
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:34.020454 30882 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:34.020727 30882 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:34.020768 30882 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:34.024736 30882 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:34.024799 30882 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:34.024817 30882 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:34.024840 30882 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:34.024855 30882 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:34.030275 30882 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42703
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30882
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:34.031296 30882 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:34.032259 30882 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:34.040686 30887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:34.040690 30890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:34.040783 30882 server_base.cc:1034] running on GCE node
W20260926 12:37:34.040678 30888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:34.041570 30882 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:34.042668 30882 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:34.043838 30882 hybrid_clock.cc:648] HybridClock initialized: now 1790426254043818 us; error 36 us; skew 500 ppm
Sep 26 12:37:34 dist-test-slave-t7tk krb5kdc[30869](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:34.046536 30882 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:37:34.047405 30882 webserver.cc:460] Webserver started at http://127.0.0.1:42741/ using document root <none> and password file <none>
I20260926 12:37:34.047925 30882 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:34.047999 30882 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:34.048202 30882 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:34.050230 30882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "804e5617126f4938bbdcdc74c1a4c3b3"
format_stamp: "Formatted at 2026-09-26 12:37:34 on dist-test-slave-t7tk"
server_key: "ce9362bb2ad9c61c19d71cfd81a62201"
server_key_iv: "8d371bc41638e953009c39da1d50f7e5"
server_key_version: "encryptionkey@0"
I20260926 12:37:34.050740 30882 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "804e5617126f4938bbdcdc74c1a4c3b3"
format_stamp: "Formatted at 2026-09-26 12:37:34 on dist-test-slave-t7tk"
server_key: "ce9362bb2ad9c61c19d71cfd81a62201"
server_key_iv: "8d371bc41638e953009c39da1d50f7e5"
server_key_version: "encryptionkey@0"
I20260926 12:37:34.054346 30882 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:37:34.056689 30897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:34.057952 30882 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:37:34.058099 30882 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "804e5617126f4938bbdcdc74c1a4c3b3"
format_stamp: "Formatted at 2026-09-26 12:37:34 on dist-test-slave-t7tk"
server_key: "ce9362bb2ad9c61c19d71cfd81a62201"
server_key_iv: "8d371bc41638e953009c39da1d50f7e5"
server_key_version: "encryptionkey@0"
I20260926 12:37:34.058192 30882 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:34.075551 30882 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:34.078717 30882 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:34.078898 30882 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:34.085489 30882 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42703
I20260926 12:37:34.085525 30949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42703 every 8 connection(s)
I20260926 12:37:34.086467 30882 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:34.088248 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 30882
I20260926 12:37:34.088362 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:34.088665 10632 external_mini_cluster.cc:1428] Setting key e4b9489100f3ec3633fd36d7ab8c082b
I20260926 12:37:34.089406 30950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:34.095777 30950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3: Bootstrap starting.
Sep 26 12:37:34 dist-test-slave-t7tk krb5kdc[30869](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 12:37:34.098800 30950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:34.099664 30950 log.cc:824] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:34.102008 30950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3: No bootstrap required, opened a new log
I20260926 12:37:34.104151 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:34.090115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43861 (local address 127.0.0.1:42703)
0926 12:37:34.090600 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:34.090617 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:34.090655 (+ 38us) server_negotiation.cc:407] Connection header received
0926 12:37:34.091447 (+ 792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:34.091530 (+ 83us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:34.091908 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:34.092268 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:34.093130 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:34.094158 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:34.095010 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:34.095663 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:34.095819 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:34.099153 (+ 3334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:34.099159 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:34.099176 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:34.099211 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:34.101195 (+ 1984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:34.101756 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:34.101762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:34.101768 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:34.101910 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:34.102211 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:34.102215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:34.102216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:34.102640 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:34.102833 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:34.103184 (+ 351us) server_negotiation.cc:299] Negotiation successful
0926 12:37:34.103397 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":161,"threads_started":1}
I20260926 12:37:34.106026 30950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804e5617126f4938bbdcdc74c1a4c3b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42703 } }
I20260926 12:37:34.106287 30950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:34.106405 30950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 804e5617126f4938bbdcdc74c1a4c3b3, State: Initialized, Role: FOLLOWER
I20260926 12:37:34.106863 30950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804e5617126f4938bbdcdc74c1a4c3b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42703 } }
I20260926 12:37:34.107023 30950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:34.107088 30950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:34.107182 30950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:34.107429 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:37:34.108139 30950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804e5617126f4938bbdcdc74c1a4c3b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42703 } }
I20260926 12:37:34.108479 30950 leader_election.cc:302] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 804e5617126f4938bbdcdc74c1a4c3b3; no voters:
I20260926 12:37:34.108821 30950 leader_election.cc:288] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:34.108904 30959 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:34.109135 30959 raft_consensus.cc:686] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3 [term 1 LEADER]: Becoming Leader. State: Replica: 804e5617126f4938bbdcdc74c1a4c3b3, State: Running, Role: LEADER
I20260926 12:37:34.109508 30959 consensus_queue.cc:237] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804e5617126f4938bbdcdc74c1a4c3b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42703 } }
I20260926 12:37:34.110249 30950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:34.110898 30959 sys_catalog.cc:455] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 804e5617126f4938bbdcdc74c1a4c3b3. Latest consensus state: current_term: 1 leader_uuid: "804e5617126f4938bbdcdc74c1a4c3b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804e5617126f4938bbdcdc74c1a4c3b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42703 } } }
I20260926 12:37:34.111125 30959 sys_catalog.cc:458] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:34.111258 30960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "804e5617126f4938bbdcdc74c1a4c3b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "804e5617126f4938bbdcdc74c1a4c3b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42703 } } }
I20260926 12:37:34.111366 30960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:34.111639 30969 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:34.115267 30969 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:34.118206 30953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:34.109624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46679 (local address 127.0.0.1:42703)
0926 12:37:34.109741 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:34.109745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:34.109756 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:37:34.109791 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:34.109793 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:34.109835 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:34.109899 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:34.110579 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:34.111331 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:34.112392 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:34.112762 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:34.112862 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:34.114658 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:34.114662 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:34.114665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:34.114691 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:34.116323 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:34.116860 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:34.116863 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:34.116866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:34.116969 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:34.117360 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:34.117364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:34.117365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:34.117566 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:34.117697 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:34.117971 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 12:37:34.118061 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":42}
I20260926 12:37:34.121667 30969 catalog_manager.cc:1347] Generated new cluster ID: be26f799e1754cfa983f54df1f891348
I20260926 12:37:34.121735 30969 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:34.137405 30969 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:34.138728 30969 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:34.144488 30969 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 804e5617126f4938bbdcdc74c1a4c3b3: Generated new TSK 0
I20260926 12:37:34.145414 30969 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:37:34.151386 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 30882
W20260926 12:37:34.156746 30867 connection.cc:535] client connection to 127.0.0.1:42703 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42703): Connection reset by peer (error 104)
2026-09-26T12:37:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1325 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:34 dist-test-slave-t7tk krb5kdc[30981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:34 dist-test-slave-t7tk krb5kdc[30981](info): set up 2 sockets
Sep 26 12:37:34 dist-test-slave-t7tk krb5kdc[30981](info): commencing operation
krb5kdc: starting...
W20260926 12:37:35.114634 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:35 dist-test-slave-t7tk krb5kdc[30981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:37:35.232463 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35713
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43569
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35713
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:35.341509 30994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:35.341807 30994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:35.341866 30994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:35.345927 30994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:35.345994 30994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:35.346021 30994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:35.346042 30994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:35.346072 30994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:35.351580 30994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43569
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35713
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:35.352730 30994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:35.353790 30994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:35.362452 31002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:35.362433 30999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:35.362437 31000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:35.362726 30994 server_base.cc:1034] running on GCE node
I20260926 12:37:35.363390 30994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:35.364568 30994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:35.365813 30994 hybrid_clock.cc:648] HybridClock initialized: now 1790426255365791 us; error 39 us; skew 500 ppm
Sep 26 12:37:35 dist-test-slave-t7tk krb5kdc[30981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:35.368549 30994 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:37:35.369541 30994 webserver.cc:460] Webserver started at http://127.0.0.1:41543/ using document root <none> and password file <none>
I20260926 12:37:35.370098 30994 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:35.370164 30994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:35.370361 30994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:35.372735 30994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "c1b86d7be0aa402f8778f746b62d3f1a"
format_stamp: "Formatted at 2026-09-26 12:37:35 on dist-test-slave-t7tk"
server_key: "7c16760ac8864758c89f1ab866f899da"
server_key_iv: "ae4a000ef10bdc2106ad93993256f621"
server_key_version: "encryptionkey@0"
I20260926 12:37:35.373342 30994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "c1b86d7be0aa402f8778f746b62d3f1a"
format_stamp: "Formatted at 2026-09-26 12:37:35 on dist-test-slave-t7tk"
server_key: "7c16760ac8864758c89f1ab866f899da"
server_key_iv: "ae4a000ef10bdc2106ad93993256f621"
server_key_version: "encryptionkey@0"
I20260926 12:37:35.377151 30994 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:37:35.379698 31009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:35.380939 30994 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:37:35.381060 30994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "c1b86d7be0aa402f8778f746b62d3f1a"
format_stamp: "Formatted at 2026-09-26 12:37:35 on dist-test-slave-t7tk"
server_key: "7c16760ac8864758c89f1ab866f899da"
server_key_iv: "ae4a000ef10bdc2106ad93993256f621"
server_key_version: "encryptionkey@0"
I20260926 12:37:35.381202 30994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:35.406128 30994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:35.409157 30994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:35.409348 30994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:35.416761 30994 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35713
I20260926 12:37:35.416815 31061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35713 every 8 connection(s)
I20260926 12:37:35.417824 30994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:35.418577 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 30994
I20260926 12:37:35.418715 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:35.419018 10632 external_mini_cluster.cc:1428] Setting key 563c5c20e2ac6d72e2b530924cd2b3f0
I20260926 12:37:35.421339 31062 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:35.427189 31062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a: Bootstrap starting.
I20260926 12:37:35.428153 31065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:35.420465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37539 (local address 127.0.0.1:35713)
0926 12:37:35.420932 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:35.420947 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:35.420984 (+ 37us) server_negotiation.cc:407] Connection header received
0926 12:37:35.421704 (+ 720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:35.421724 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:35.422091 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:35.422421 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:35.422875 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:35.423899 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:35.424835 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:35.425349 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:35.425492 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:35.426440 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:35.426445 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:35.426461 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:35.426857 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:35.426913 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:35.426936 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:35.427050 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:35.427262 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 12:37:35.427405 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":154,"threads_started":1}
I20260926 12:37:35.429589 31062 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:35.430416 31062 log.cc:824] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:35.431211 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:37:35.432469 31062 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a: No bootstrap required, opened a new log
I20260926 12:37:35.437192 31062 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b86d7be0aa402f8778f746b62d3f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35713 } }
I20260926 12:37:35.437563 31062 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:35.437598 31065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:35.433120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37311 (local address 127.0.0.1:35713)
0926 12:37:35.433213 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:35.433216 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:35.433499 (+ 283us) server_negotiation.cc:407] Connection header received
0926 12:37:35.433644 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:35.433647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:35.433687 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:35.433749 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:35.434066 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:35.434586 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:35.435841 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:35.436077 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:35.436155 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:35.437080 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:35.437083 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:35.437085 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:35.437127 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:35.437156 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:35.437168 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:35.437237 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:35.437391 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 12:37:35.437425 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 12:37:35.437633 31062 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1b86d7be0aa402f8778f746b62d3f1a, State: Initialized, Role: FOLLOWER
I20260926 12:37:35.438083 31062 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b86d7be0aa402f8778f746b62d3f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35713 } }
I20260926 12:37:35.438223 31062 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:35.438289 31062 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:35.438387 31062 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:35.439322 31062 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b86d7be0aa402f8778f746b62d3f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35713 } }
I20260926 12:37:35.439654 31062 leader_election.cc:302] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1b86d7be0aa402f8778f746b62d3f1a; no voters:
I20260926 12:37:35.439954 31062 leader_election.cc:288] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:35.440104 31072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:35.440303 31072 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a [term 1 LEADER]: Becoming Leader. State: Replica: c1b86d7be0aa402f8778f746b62d3f1a, State: Running, Role: LEADER
I20260926 12:37:35.440626 31072 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b86d7be0aa402f8778f746b62d3f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35713 } }
I20260926 12:37:35.441169 31062 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:35.442276 31073 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1b86d7be0aa402f8778f746b62d3f1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b86d7be0aa402f8778f746b62d3f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35713 } } }
I20260926 12:37:35.442278 31074 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1b86d7be0aa402f8778f746b62d3f1a. Latest consensus state: current_term: 1 leader_uuid: "c1b86d7be0aa402f8778f746b62d3f1a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1b86d7be0aa402f8778f746b62d3f1a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35713 } } }
I20260926 12:37:35.442466 31073 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:35.442518 31074 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:35.442862 31083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:35.446209 31083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:35.452437 31083 catalog_manager.cc:1347] Generated new cluster ID: 3602360e9493491b88affa76811b61f1
I20260926 12:37:35.452525 31083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:35.486286 31083 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:35.487254 31083 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:35.491219 31083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1b86d7be0aa402f8778f746b62d3f1a: Generated new TSK 0
I20260926 12:37:35.492027 31083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:37:35.499754 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 30994
W20260926 12:37:35.504429 30979 connection.cc:535] client connection to 127.0.0.1:35713 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35713): Connection reset by peer (error 104)
2026-09-26T12:37:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1347 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:35 dist-test-slave-t7tk krb5kdc[31093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:35 dist-test-slave-t7tk krb5kdc[31093](info): set up 2 sockets
Sep 26 12:37:35 dist-test-slave-t7tk krb5kdc[31093](info): commencing operation
krb5kdc: starting...
W20260926 12:37:36.470573 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:36 dist-test-slave-t7tk krb5kdc[31093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:37:36.592080 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43797
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:36.700227 31106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:36.700531 31106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:36.700575 31106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:36.704633 31106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:36.704703 31106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:36.704721 31106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:36.704737 31106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:36.704751 31106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:36.710048 31106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43797
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:36.711133 31106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:36.712138 31106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:36.720561 31114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:36.720520 31111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:36.720546 31112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:36.720688 31106 server_base.cc:1034] running on GCE node
I20260926 12:37:36.721468 31106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:36.722538 31106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:36.723694 31106 hybrid_clock.cc:648] HybridClock initialized: now 1790426256723679 us; error 27 us; skew 500 ppm
Sep 26 12:37:36 dist-test-slave-t7tk krb5kdc[31093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:36.726478 31106 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:37:36.727381 31106 webserver.cc:460] Webserver started at http://127.0.0.1:33957/ using document root <none> and password file <none>
I20260926 12:37:36.727921 31106 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:36.727986 31106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:36.728173 31106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:36.730218 31106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "bf351e30fb8e457bb1e535630367c87f"
format_stamp: "Formatted at 2026-09-26 12:37:36 on dist-test-slave-t7tk"
server_key: "ecce916c796b2df8f0a160ad60e09888"
server_key_iv: "c461d28d279ae55c0bd8c39459575668"
server_key_version: "encryptionkey@0"
I20260926 12:37:36.730765 31106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "bf351e30fb8e457bb1e535630367c87f"
format_stamp: "Formatted at 2026-09-26 12:37:36 on dist-test-slave-t7tk"
server_key: "ecce916c796b2df8f0a160ad60e09888"
server_key_iv: "c461d28d279ae55c0bd8c39459575668"
server_key_version: "encryptionkey@0"
I20260926 12:37:36.734508 31106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:37:36.737071 31121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:36.738430 31106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:37:36.738552 31106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "bf351e30fb8e457bb1e535630367c87f"
format_stamp: "Formatted at 2026-09-26 12:37:36 on dist-test-slave-t7tk"
server_key: "ecce916c796b2df8f0a160ad60e09888"
server_key_iv: "c461d28d279ae55c0bd8c39459575668"
server_key_version: "encryptionkey@0"
I20260926 12:37:36.738657 31106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:36.743400 31106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:36.746770 31106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:36.746974 31106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:36.754117 31106 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43797
I20260926 12:37:36.754204 31173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43797 every 8 connection(s)
I20260926 12:37:36.755116 31106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:36.757913 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 31106
I20260926 12:37:36.758028 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:36.758093 31174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:36.758323 10632 external_mini_cluster.cc:1428] Setting key c6e4bb46534107d2da8b4a874acab2a2
I20260926 12:37:36.763849 31177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:36.759736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51863 (local address 127.0.0.1:43797)
0926 12:37:36.760203 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:36.760219 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:36.760256 (+ 37us) server_negotiation.cc:407] Connection header received
0926 12:37:36.761256 (+ 1000us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:36.761275 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:36.761582 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:36.761902 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:36.762685 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:36.762690 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:36.762702 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:36.762713 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:36.762767 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:36.762787 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:36.762910 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:36.762991 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 12:37:36.763167 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":137,"threads_started":1}
I20260926 12:37:36.764662 31174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f: Bootstrap starting.
I20260926 12:37:36.766750 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:37:36.767206 31174 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:36.767948 31174 log.cc:824] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:36.770156 31177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:36.768654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58051 (local address 127.0.0.1:43797)
0926 12:37:36.768775 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:36.768778 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:36.768896 (+ 118us) server_negotiation.cc:407] Connection header received
0926 12:37:36.769009 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:36.769011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:36.769039 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:36.769148 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:36.769809 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:36.769811 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:36.769814 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:36.769823 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:36.769847 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:36.769857 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:36.769923 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:36.770025 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 12:37:36.770057 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 12:37:36.770466 31174 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f: No bootstrap required, opened a new log
I20260926 12:37:36.774024 31174 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf351e30fb8e457bb1e535630367c87f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43797 } }
I20260926 12:37:36.774260 31174 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:36.774293 31174 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf351e30fb8e457bb1e535630367c87f, State: Initialized, Role: FOLLOWER
I20260926 12:37:36.774700 31174 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf351e30fb8e457bb1e535630367c87f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43797 } }
I20260926 12:37:36.774829 31174 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:36.774870 31174 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:36.774943 31174 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:36.775816 31174 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf351e30fb8e457bb1e535630367c87f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43797 } }
I20260926 12:37:36.776121 31174 leader_election.cc:302] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf351e30fb8e457bb1e535630367c87f; no voters:
I20260926 12:37:36.776381 31174 leader_election.cc:288] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:36.776503 31184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:36.776753 31184 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f [term 1 LEADER]: Becoming Leader. State: Replica: bf351e30fb8e457bb1e535630367c87f, State: Running, Role: LEADER
I20260926 12:37:36.777132 31184 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf351e30fb8e457bb1e535630367c87f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43797 } }
I20260926 12:37:36.777690 31174 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:36.778823 31185 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf351e30fb8e457bb1e535630367c87f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf351e30fb8e457bb1e535630367c87f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43797 } } }
I20260926 12:37:36.778959 31185 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:36.779292 31186 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf351e30fb8e457bb1e535630367c87f. Latest consensus state: current_term: 1 leader_uuid: "bf351e30fb8e457bb1e535630367c87f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf351e30fb8e457bb1e535630367c87f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43797 } } }
I20260926 12:37:36.779422 31186 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:36.779348 31193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:36.782310 31193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:36.787611 31193 catalog_manager.cc:1347] Generated new cluster ID: 28c3d5a3c9274b16b7017dd298b9f958
I20260926 12:37:36.787686 31193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:36.798503 31193 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:36.798832 31193 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:36.807717 31193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf351e30fb8e457bb1e535630367c87f: Generated new TSK 0
I20260926 12:37:36.808441 31193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:37:36.818871 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 31106
2026-09-26T12:37:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1318 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:36 dist-test-slave-t7tk krb5kdc[31205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:36 dist-test-slave-t7tk krb5kdc[31205](info): set up 2 sockets
Sep 26 12:37:36 dist-test-slave-t7tk krb5kdc[31205](info): commencing operation
krb5kdc: starting...
W20260926 12:37:37.786561 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.949s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:37 dist-test-slave-t7tk krb5kdc[31205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:37:37.908463 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45853
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:38.017071 31218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:38.017377 31218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:38.017419 31218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:38.021420 31218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:38.021482 31218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:38.021500 31218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:38.021515 31218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:38.021530 31218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:38.026834 31218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:38.027915 31218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:38.029271 31218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:38.037783 31223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:38.037842 31218 server_base.cc:1034] running on GCE node
W20260926 12:37:38.037753 31226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:38.037741 31224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:38.038596 31218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:38.039702 31218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:38.040879 31218 hybrid_clock.cc:648] HybridClock initialized: now 1790426258040850 us; error 45 us; skew 500 ppm
Sep 26 12:37:38 dist-test-slave-t7tk krb5kdc[31205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426258, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:38.049470 31218 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:37:38.050493 31218 webserver.cc:460] Webserver started at http://127.0.0.1:44279/ using document root <none> and password file <none>
I20260926 12:37:38.051065 31218 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:38.051131 31218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:38.051324 31218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:38.053455 31218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "0bbfc4bbc06c44b6a486dd988df69a89"
format_stamp: "Formatted at 2026-09-26 12:37:38 on dist-test-slave-t7tk"
server_key: "9f240cdc48bf1e028af7824471d71e37"
server_key_iv: "b4bcdbcbbdb5451fc01a3bf0bb286316"
server_key_version: "encryptionkey@0"
I20260926 12:37:38.054009 31218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "0bbfc4bbc06c44b6a486dd988df69a89"
format_stamp: "Formatted at 2026-09-26 12:37:38 on dist-test-slave-t7tk"
server_key: "9f240cdc48bf1e028af7824471d71e37"
server_key_iv: "b4bcdbcbbdb5451fc01a3bf0bb286316"
server_key_version: "encryptionkey@0"
I20260926 12:37:38.057920 31218 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:37:38.060315 31233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:38.061784 31218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:37:38.061901 31218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "0bbfc4bbc06c44b6a486dd988df69a89"
format_stamp: "Formatted at 2026-09-26 12:37:38 on dist-test-slave-t7tk"
server_key: "9f240cdc48bf1e028af7824471d71e37"
server_key_iv: "b4bcdbcbbdb5451fc01a3bf0bb286316"
server_key_version: "encryptionkey@0"
I20260926 12:37:38.062008 31218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:38.080992 31218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:38.084481 31218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:38.084698 31218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:38.091773 31218 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45853
I20260926 12:37:38.091800 31285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45853 every 8 connection(s)
I20260926 12:37:38.092741 31218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:38.094836 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 31218
I20260926 12:37:38.094971 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:38.095237 10632 external_mini_cluster.cc:1428] Setting key b50e26f662953428a0dda86e5bfd341d
I20260926 12:37:38.095871 31286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:38.102924 31286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89: Bootstrap starting.
Sep 26 12:37:38 dist-test-slave-t7tk krb5kdc[31205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426257, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 12:37:38.105449 31286 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:38.106389 31286 log.cc:824] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:38.108692 31286 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89: No bootstrap required, opened a new log
I20260926 12:37:38.110967 31289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:38.096717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36611 (local address 127.0.0.1:45853)
0926 12:37:38.097184 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:38.097209 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:38.097254 (+ 45us) server_negotiation.cc:407] Connection header received
0926 12:37:38.098038 (+ 784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:38.098062 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:38.098515 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:38.098899 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:38.099583 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:38.101036 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:38.102001 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:38.102681 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:38.102841 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:38.105575 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:38.105581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:38.105599 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:38.105634 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:38.107540 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:38.108007 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:38.108013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:38.108019 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:38.108157 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:38.108400 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:38.108403 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:38.108405 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:38.108920 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:38.109149 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:38.109492 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 12:37:38.109960 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":130,"threads_started":1}
I20260926 12:37:38.114132 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:37:38.114068 31286 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbfc4bbc06c44b6a486dd988df69a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45853 } }
I20260926 12:37:38.114317 31286 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:38.114375 31286 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bbfc4bbc06c44b6a486dd988df69a89, State: Initialized, Role: FOLLOWER
I20260926 12:37:38.114785 31286 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbfc4bbc06c44b6a486dd988df69a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45853 } }
I20260926 12:37:38.114902 31286 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:38.114957 31286 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:38.115051 31286 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:38.116004 31286 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbfc4bbc06c44b6a486dd988df69a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45853 } }
I20260926 12:37:38.116338 31286 leader_election.cc:302] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0bbfc4bbc06c44b6a486dd988df69a89; no voters:
I20260926 12:37:38.116641 31286 leader_election.cc:288] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:38.116727 31296 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:38.116881 31296 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89 [term 1 LEADER]: Becoming Leader. State: Replica: 0bbfc4bbc06c44b6a486dd988df69a89, State: Running, Role: LEADER
I20260926 12:37:38.117295 31296 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbfc4bbc06c44b6a486dd988df69a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45853 } }
I20260926 12:37:38.117995 31286 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:38.118547 31296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0bbfc4bbc06c44b6a486dd988df69a89. Latest consensus state: current_term: 1 leader_uuid: "0bbfc4bbc06c44b6a486dd988df69a89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbfc4bbc06c44b6a486dd988df69a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45853 } } }
I20260926 12:37:38.118673 31296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:38.119120 31302 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:38.119305 31297 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0bbfc4bbc06c44b6a486dd988df69a89" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0bbfc4bbc06c44b6a486dd988df69a89" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45853 } } }
I20260926 12:37:38.119433 31297 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:38.122527 31302 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:38.124837 31289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:38.116442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54843 (local address 127.0.0.1:45853)
0926 12:37:38.116567 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:38.116571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:38.116583 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:37:38.116617 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:38.116620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:38.116670 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:38.116734 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:38.117580 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:38.118244 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:38.119348 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:38.119666 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:38.119743 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:38.121430 (+ 1687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:38.121434 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:38.121437 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:38.121474 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:38.123059 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:38.123553 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:38.123556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:38.123558 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:38.123648 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:38.123948 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:38.123951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:38.123953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:38.124176 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:38.124307 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:38.124563 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 12:37:38.124685 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:37:38.128600 31302 catalog_manager.cc:1347] Generated new cluster ID: fd946402734d4c56a9b6f875cb4b6ac3
I20260926 12:37:38.128659 31302 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:38.143028 31302 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:38.144029 31302 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:38.154208 31302 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0bbfc4bbc06c44b6a486dd988df69a89: Generated new TSK 0
I20260926 12:37:38.155067 31302 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:37:38.161238 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 31218
2026-09-26T12:37:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:38 dist-test-slave-t7tk krb5kdc[31317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:38 dist-test-slave-t7tk krb5kdc[31317](info): set up 2 sockets
Sep 26 12:37:38 dist-test-slave-t7tk krb5kdc[31317](info): commencing operation
krb5kdc: starting...
W20260926 12:37:39.134616 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.954s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:39 dist-test-slave-t7tk krb5kdc[31317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:37:39.256672 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42557
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:39.366755 31330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:39.367026 31330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:39.367067 31330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:39.371075 31330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:39.371141 31330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:39.371160 31330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:39.371174 31330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:39.371188 31330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:39.376540 31330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:39.377632 31330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:39.378712 31330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:39.387219 31335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:39.387239 31336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:39.387243 31338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:39.387302 31330 server_base.cc:1034] running on GCE node
I20260926 12:37:39.388052 31330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:39.389174 31330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:39.390345 31330 hybrid_clock.cc:648] HybridClock initialized: now 1790426259390327 us; error 35 us; skew 500 ppm
Sep 26 12:37:39 dist-test-slave-t7tk krb5kdc[31317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:39.393078 31330 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:37:39.393992 31330 webserver.cc:460] Webserver started at http://127.0.0.1:34765/ using document root <none> and password file <none>
I20260926 12:37:39.394532 31330 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:39.394595 31330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:39.394781 31330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:39.396739 31330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "dfcc5efaa6e74ad18b14e5cf7bca69a5"
format_stamp: "Formatted at 2026-09-26 12:37:39 on dist-test-slave-t7tk"
server_key: "1fb4e21562cbd2a46e8a51a180aba215"
server_key_iv: "189aa01ce8249dd1b9d8e11bf334a399"
server_key_version: "encryptionkey@0"
I20260926 12:37:39.397289 31330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "dfcc5efaa6e74ad18b14e5cf7bca69a5"
format_stamp: "Formatted at 2026-09-26 12:37:39 on dist-test-slave-t7tk"
server_key: "1fb4e21562cbd2a46e8a51a180aba215"
server_key_iv: "189aa01ce8249dd1b9d8e11bf334a399"
server_key_version: "encryptionkey@0"
I20260926 12:37:39.400758 31330 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260926 12:37:39.403139 31345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:39.404444 31330 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:37:39.404574 31330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "dfcc5efaa6e74ad18b14e5cf7bca69a5"
format_stamp: "Formatted at 2026-09-26 12:37:39 on dist-test-slave-t7tk"
server_key: "1fb4e21562cbd2a46e8a51a180aba215"
server_key_iv: "189aa01ce8249dd1b9d8e11bf334a399"
server_key_version: "encryptionkey@0"
I20260926 12:37:39.404672 31330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:39.423570 31330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:39.426755 31330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:39.426946 31330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:39.434068 31330 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42557
I20260926 12:37:39.434338 31397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42557 every 8 connection(s)
I20260926 12:37:39.435089 31330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:39.437980 31398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:39.442950 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 31330
I20260926 12:37:39.443125 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:39.443425 10632 external_mini_cluster.cc:1428] Setting key 359ec83f48e1f88e44a07b8baa81883f
I20260926 12:37:39.443893 31398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5: Bootstrap starting.
I20260926 12:37:39.446383 31398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:39.447098 31398 log.cc:824] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:39.448944 31398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5: No bootstrap required, opened a new log
I20260926 12:37:39.452359 31401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:39.444821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42213 (local address 127.0.0.1:42557)
0926 12:37:39.445358 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:39.445374 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:39.445411 (+ 37us) server_negotiation.cc:407] Connection header received
0926 12:37:39.446115 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:39.446133 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:39.446501 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:39.446862 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:39.447321 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:39.448295 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:39.449248 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:39.449741 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:39.449886 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:39.450678 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:39.450683 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:39.450696 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:39.451078 (+ 382us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:39.451133 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:39.451156 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:39.451239 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:39.451394 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 12:37:39.451605 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":177,"threads_started":1}
I20260926 12:37:39.452443 31398 raft_consensus.cc:348] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcc5efaa6e74ad18b14e5cf7bca69a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42557 } }
I20260926 12:37:39.452740 31398 raft_consensus.cc:374] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:39.452796 31398 raft_consensus.cc:729] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfcc5efaa6e74ad18b14e5cf7bca69a5, State: Initialized, Role: FOLLOWER
I20260926 12:37:39.453263 31398 consensus_queue.cc:260] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcc5efaa6e74ad18b14e5cf7bca69a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42557 } }
I20260926 12:37:39.453428 31398 raft_consensus.cc:388] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:39.453496 31398 raft_consensus.cc:482] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:39.453593 31398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:39.454569 31398 raft_consensus.cc:504] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcc5efaa6e74ad18b14e5cf7bca69a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42557 } }
I20260926 12:37:39.454916 31398 leader_election.cc:302] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dfcc5efaa6e74ad18b14e5cf7bca69a5; no voters:
I20260926 12:37:39.455240 31398 leader_election.cc:288] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:39.455327 31403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:39.455446 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:37:39.455538 31403 raft_consensus.cc:686] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5 [term 1 LEADER]: Becoming Leader. State: Replica: dfcc5efaa6e74ad18b14e5cf7bca69a5, State: Running, Role: LEADER
I20260926 12:37:39.455832 31403 consensus_queue.cc:237] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcc5efaa6e74ad18b14e5cf7bca69a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42557 } }
I20260926 12:37:39.456576 31398 sys_catalog.cc:564] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:39.457259 31404 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dfcc5efaa6e74ad18b14e5cf7bca69a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcc5efaa6e74ad18b14e5cf7bca69a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42557 } } }
I20260926 12:37:39.457379 31404 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:39.457686 31405 sys_catalog.cc:455] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dfcc5efaa6e74ad18b14e5cf7bca69a5. Latest consensus state: current_term: 1 leader_uuid: "dfcc5efaa6e74ad18b14e5cf7bca69a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dfcc5efaa6e74ad18b14e5cf7bca69a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42557 } } }
I20260926 12:37:39.457763 31405 sys_catalog.cc:458] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:39.457793 31416 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:39.461020 31416 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:39.462826 31401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:39.458211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33367 (local address 127.0.0.1:42557)
0926 12:37:39.458312 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:39.458316 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:39.458919 (+ 603us) server_negotiation.cc:407] Connection header received
0926 12:37:39.459108 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:39.459111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:39.459159 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:39.459245 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:39.459532 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:39.460025 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:39.461216 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:39.461514 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:39.461610 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:39.462427 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:39.462430 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:39.462433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:39.462445 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:39.462486 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:39.462497 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:39.462556 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:39.462648 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 12:37:39.462697 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 12:37:39.466938 31416 catalog_manager.cc:1347] Generated new cluster ID: a91ae1708c14498cbec65c7d16cf0788
I20260926 12:37:39.467008 31416 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:39.490092 31416 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:39.491103 31416 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:39.501394 31416 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dfcc5efaa6e74ad18b14e5cf7bca69a5: Generated new TSK 0
I20260926 12:37:39.502148 31416 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:37:39.510210 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 31330
2026-09-26T12:37:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:39 dist-test-slave-t7tk krb5kdc[31429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:39 dist-test-slave-t7tk krb5kdc[31429](info): set up 2 sockets
Sep 26 12:37:39 dist-test-slave-t7tk krb5kdc[31429](info): commencing operation
krb5kdc: starting...
W20260926 12:37:40.467123 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:40 dist-test-slave-t7tk krb5kdc[31429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426260, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:37:40.593587 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46437
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46437
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:40.703507 31442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:40.703807 31442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:40.703868 31442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:40.708005 31442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:40.708086 31442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:40.708112 31442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:40.708133 31442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:40.708153 31442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:40.713618 31442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46437
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:40.714709 31442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:40.715746 31442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:37:40.724519 31442 server_base.cc:1034] running on GCE node
W20260926 12:37:40.724367 31448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:40.724367 31450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:40.724440 31447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:40.725302 31442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:40.726401 31442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:40.727577 31442 hybrid_clock.cc:648] HybridClock initialized: now 1790426260727553 us; error 37 us; skew 500 ppm
Sep 26 12:37:40 dist-test-slave-t7tk krb5kdc[31429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426260, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:40.730484 31442 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:37:40.731341 31442 webserver.cc:460] Webserver started at http://127.0.0.1:35837/ using document root <none> and password file <none>
I20260926 12:37:40.731844 31442 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:40.731909 31442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:40.732080 31442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:40.734109 31442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "4ecaf2b2ad2746c8a3008c38a1aa3c3a"
format_stamp: "Formatted at 2026-09-26 12:37:40 on dist-test-slave-t7tk"
server_key: "ce7fe3b11f65564faaab06d94876385c"
server_key_iv: "24bf2dc9723b16254645b7045a6afa29"
server_key_version: "encryptionkey@0"
I20260926 12:37:40.734614 31442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "4ecaf2b2ad2746c8a3008c38a1aa3c3a"
format_stamp: "Formatted at 2026-09-26 12:37:40 on dist-test-slave-t7tk"
server_key: "ce7fe3b11f65564faaab06d94876385c"
server_key_iv: "24bf2dc9723b16254645b7045a6afa29"
server_key_version: "encryptionkey@0"
I20260926 12:37:40.738201 31442 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:37:40.740527 31457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:40.741864 31442 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:37:40.741994 31442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "4ecaf2b2ad2746c8a3008c38a1aa3c3a"
format_stamp: "Formatted at 2026-09-26 12:37:40 on dist-test-slave-t7tk"
server_key: "ce7fe3b11f65564faaab06d94876385c"
server_key_iv: "24bf2dc9723b16254645b7045a6afa29"
server_key_version: "encryptionkey@0"
I20260926 12:37:40.742100 31442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:40.746721 31442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:40.749905 31442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:40.750079 31442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:40.756740 31442 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46437
I20260926 12:37:40.756757 31509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46437 every 8 connection(s)
I20260926 12:37:40.757712 31442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:40.759673 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 31442
I20260926 12:37:40.759794 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:40.760076 10632 external_mini_cluster.cc:1428] Setting key e455c99b354f7c6580812cf3625c1276
I20260926 12:37:40.760937 31510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:40.765599 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:40.761489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36267 (local address 127.0.0.1:46437)
0926 12:37:40.761957 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:40.761972 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:40.762191 (+ 219us) server_negotiation.cc:407] Connection header received
0926 12:37:40.762918 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:40.762935 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:40.763269 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:40.763560 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:40.764351 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:40.764355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:40.764367 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:40.764376 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:40.764427 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:40.764445 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:40.764561 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:40.764672 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 12:37:40.764819 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":145,"threads_started":1}
I20260926 12:37:40.767599 31510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a: Bootstrap starting.
I20260926 12:37:40.768393 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:37:40.769943 31510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:40.770702 31510 log.cc:824] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:40.771917 31513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:40.770351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55729 (local address 127.0.0.1:46437)
0926 12:37:40.770477 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:40.770481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:40.770722 (+ 241us) server_negotiation.cc:407] Connection header received
0926 12:37:40.770824 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:40.770827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:40.770861 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:40.770924 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:40.771572 (+ 648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:40.771576 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:40.771578 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:40.771590 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:40.771613 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:40.771623 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:40.771691 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:40.771768 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 12:37:40.771805 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:37:40.772810 31510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a: No bootstrap required, opened a new log
I20260926 12:37:40.776329 31510 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecaf2b2ad2746c8a3008c38a1aa3c3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46437 } }
I20260926 12:37:40.776592 31510 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:40.776652 31510 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecaf2b2ad2746c8a3008c38a1aa3c3a, State: Initialized, Role: FOLLOWER
I20260926 12:37:40.777087 31510 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecaf2b2ad2746c8a3008c38a1aa3c3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46437 } }
I20260926 12:37:40.777278 31510 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:40.777346 31510 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:40.777447 31510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:40.778409 31510 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecaf2b2ad2746c8a3008c38a1aa3c3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46437 } }
I20260926 12:37:40.778798 31510 leader_election.cc:302] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ecaf2b2ad2746c8a3008c38a1aa3c3a; no voters:
I20260926 12:37:40.779147 31510 leader_election.cc:288] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:40.779265 31520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:40.779484 31520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a [term 1 LEADER]: Becoming Leader. State: Replica: 4ecaf2b2ad2746c8a3008c38a1aa3c3a, State: Running, Role: LEADER
I20260926 12:37:40.779799 31520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecaf2b2ad2746c8a3008c38a1aa3c3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46437 } }
I20260926 12:37:40.780359 31510 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:40.781391 31522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ecaf2b2ad2746c8a3008c38a1aa3c3a. Latest consensus state: current_term: 1 leader_uuid: "4ecaf2b2ad2746c8a3008c38a1aa3c3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecaf2b2ad2746c8a3008c38a1aa3c3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46437 } } }
I20260926 12:37:40.781520 31522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:40.781772 31521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ecaf2b2ad2746c8a3008c38a1aa3c3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ecaf2b2ad2746c8a3008c38a1aa3c3a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46437 } } }
I20260926 12:37:40.781888 31529 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:40.781924 31521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:40.784663 31529 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:40.789719 31529 catalog_manager.cc:1347] Generated new cluster ID: 7173a150c95b499789800f2689faeb19
I20260926 12:37:40.789793 31529 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:40.804909 31529 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:40.805311 31529 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:40.813308 31529 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ecaf2b2ad2746c8a3008c38a1aa3c3a: Generated new TSK 0
I20260926 12:37:40.814072 31529 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:37:40.820983 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 31442
2026-09-26T12:37:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:40 dist-test-slave-t7tk krb5kdc[31541](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:40 dist-test-slave-t7tk krb5kdc[31541](info): set up 2 sockets
Sep 26 12:37:40 dist-test-slave-t7tk krb5kdc[31541](info): commencing operation
krb5kdc: starting...
W20260926 12:37:41.778806 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:41 dist-test-slave-t7tk krb5kdc[31541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:37:41.902565 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42423
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:42.012770 31554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:42.013047 31554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:42.013090 31554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:42.017141 31554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:42.017221 31554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:42.017239 31554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:42.017256 31554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:42.017269 31554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:42.022673 31554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:42.023761 31554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:42.024756 31554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:42.033459 31559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:42.033449 31560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:42.033687 31562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:42.033779 31554 server_base.cc:1034] running on GCE node
I20260926 12:37:42.034499 31554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:42.035687 31554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:42.036876 31554 hybrid_clock.cc:648] HybridClock initialized: now 1790426262036852 us; error 35 us; skew 500 ppm
Sep 26 12:37:42 dist-test-slave-t7tk krb5kdc[31541](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426262, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:42.039834 31554 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:37:42.040760 31554 webserver.cc:460] Webserver started at http://127.0.0.1:40301/ using document root <none> and password file <none>
I20260926 12:37:42.041361 31554 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:42.041433 31554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:42.041639 31554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:42.043761 31554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "d3e1dd3f9b8f47a09ebb34399931952a"
format_stamp: "Formatted at 2026-09-26 12:37:42 on dist-test-slave-t7tk"
server_key: "124b4df60fd0eff8bdbd9c6bdb7151bd"
server_key_iv: "250a1ce206f633bc25f0b2785f29ed95"
server_key_version: "encryptionkey@0"
I20260926 12:37:42.044315 31554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "d3e1dd3f9b8f47a09ebb34399931952a"
format_stamp: "Formatted at 2026-09-26 12:37:42 on dist-test-slave-t7tk"
server_key: "124b4df60fd0eff8bdbd9c6bdb7151bd"
server_key_iv: "250a1ce206f633bc25f0b2785f29ed95"
server_key_version: "encryptionkey@0"
I20260926 12:37:42.048256 31554 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:37:42.050832 31569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:42.052343 31554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:37:42.052459 31554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "d3e1dd3f9b8f47a09ebb34399931952a"
format_stamp: "Formatted at 2026-09-26 12:37:42 on dist-test-slave-t7tk"
server_key: "124b4df60fd0eff8bdbd9c6bdb7151bd"
server_key_iv: "250a1ce206f633bc25f0b2785f29ed95"
server_key_version: "encryptionkey@0"
I20260926 12:37:42.052614 31554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:42.090322 31554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:42.093541 31554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:42.093729 31554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:42.100492 31554 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42423
I20260926 12:37:42.100638 31621 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42423 every 8 connection(s)
I20260926 12:37:42.101517 31554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:42.104393 31622 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:42.109491 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 31554
I20260926 12:37:42.109632 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:42.109895 10632 external_mini_cluster.cc:1428] Setting key 386167dc25fac5d29797b641f15b7b97
I20260926 12:37:42.110301 31622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a: Bootstrap starting.
I20260926 12:37:42.113184 31622 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:42.114120 31622 log.cc:824] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:42.116350 31622 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a: No bootstrap required, opened a new log
Sep 26 12:37:42 dist-test-slave-t7tk krb5kdc[31541](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 12:37:42.120460 31622 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e1dd3f9b8f47a09ebb34399931952a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } }
I20260926 12:37:42.120767 31622 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:42.120829 31622 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3e1dd3f9b8f47a09ebb34399931952a, State: Initialized, Role: FOLLOWER
I20260926 12:37:42.121366 31622 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e1dd3f9b8f47a09ebb34399931952a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } }
I20260926 12:37:42.121533 31622 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:42.121608 31622 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:42.121711 31622 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:42.122584 31622 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e1dd3f9b8f47a09ebb34399931952a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } }
I20260926 12:37:42.122967 31622 leader_election.cc:302] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3e1dd3f9b8f47a09ebb34399931952a; no voters:
I20260926 12:37:42.123359 31622 leader_election.cc:288] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:42.123458 31627 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:42.123667 31627 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a [term 1 LEADER]: Becoming Leader. State: Replica: d3e1dd3f9b8f47a09ebb34399931952a, State: Running, Role: LEADER
I20260926 12:37:42.124012 31627 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e1dd3f9b8f47a09ebb34399931952a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } }
I20260926 12:37:42.124739 31622 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:42.125514 31627 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3e1dd3f9b8f47a09ebb34399931952a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e1dd3f9b8f47a09ebb34399931952a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } } }
I20260926 12:37:42.125696 31627 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:42.125921 31627 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3e1dd3f9b8f47a09ebb34399931952a. Latest consensus state: current_term: 1 leader_uuid: "d3e1dd3f9b8f47a09ebb34399931952a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3e1dd3f9b8f47a09ebb34399931952a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42423 } } }
I20260926 12:37:42.126065 31627 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:42.126427 31636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:42.126798 31625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:42.111421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:39517 (local address 127.0.0.1:42423)
0926 12:37:42.111877 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:42.111896 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:42.111941 (+ 45us) server_negotiation.cc:407] Connection header received
0926 12:37:42.112775 (+ 834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:42.112798 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:42.113289 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:42.113694 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:42.114467 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:42.115845 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:42.116861 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:42.117595 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:42.117756 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:42.120595 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:42.120601 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:42.120619 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:42.120653 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:42.122568 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:42.123257 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:42.123264 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:42.123270 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:42.123428 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:42.123897 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:42.123900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:42.123902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:42.124443 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:42.125289 (+ 846us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:42.125471 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 12:37:42.125714 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":116,"threads_started":1}
I20260926 12:37:42.130091 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:37:42.131139 31636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:42.137370 31636 catalog_manager.cc:1347] Generated new cluster ID: 571514ec3fd048e38421df5c4ea76a71
I20260926 12:37:42.137449 31636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:42.144945 31625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:42.133323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:37949 (local address 127.0.0.1:42423)
0926 12:37:42.133434 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:42.133438 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:42.136062 (+ 2624us) server_negotiation.cc:407] Connection header received
0926 12:37:42.136201 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:42.136204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:42.136258 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:42.136333 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:42.137151 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:42.137879 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:42.139117 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:42.139534 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:42.139650 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:42.141365 (+ 1715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:42.141369 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:42.141372 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:42.141400 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:42.143054 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:42.143627 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:42.143631 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:42.143633 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:42.143720 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:42.144033 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:42.144036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:42.144037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:42.144240 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:42.144405 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:42.144664 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 12:37:42.144765 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 12:37:42.148656 31636 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:42.149986 31636 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:42.156209 31636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3e1dd3f9b8f47a09ebb34399931952a: Generated new TSK 0
I20260926 12:37:42.157152 31636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:37:42.164698 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 31554
W20260926 12:37:42.169564 31539 connection.cc:535] client connection to 127.0.0.1:42423 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42423): Connection reset by peer (error 104)
2026-09-26T12:37:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:42 dist-test-slave-t7tk krb5kdc[31653](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:42 dist-test-slave-t7tk krb5kdc[31653](info): set up 2 sockets
Sep 26 12:37:42 dist-test-slave-t7tk krb5kdc[31653](info): commencing operation
krb5kdc: starting...
W20260926 12:37:43.126847 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:43 dist-test-slave-t7tk krb5kdc[31653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426263, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:37:43.260053 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39119
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:43.373588 31666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:43.373876 31666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:43.373921 31666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:43.377977 31666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:43.378051 31666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:43.378072 31666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:43.378091 31666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:43.378108 31666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:43.383535 31666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:43.384619 31666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:43.385759 31666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:43.394505 31671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:43.394687 31666 server_base.cc:1034] running on GCE node
W20260926 12:37:43.394567 31672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:43.394567 31674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:43.395426 31666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:43.396602 31666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:43.397766 31666 hybrid_clock.cc:648] HybridClock initialized: now 1790426263397743 us; error 41 us; skew 500 ppm
Sep 26 12:37:43 dist-test-slave-t7tk krb5kdc[31653](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426263, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:43.406843 31666 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:37:43.407847 31666 webserver.cc:460] Webserver started at http://127.0.0.1:39023/ using document root <none> and password file <none>
I20260926 12:37:43.408398 31666 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:43.408463 31666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:43.408663 31666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:43.410777 31666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "575238d3efc442b7a7a76228530abb9d"
format_stamp: "Formatted at 2026-09-26 12:37:43 on dist-test-slave-t7tk"
server_key: "0dc393b447939971144014c1f135c05e"
server_key_iv: "30b04d0fe094e28f00ca764199bfe510"
server_key_version: "encryptionkey@0"
I20260926 12:37:43.411329 31666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "575238d3efc442b7a7a76228530abb9d"
format_stamp: "Formatted at 2026-09-26 12:37:43 on dist-test-slave-t7tk"
server_key: "0dc393b447939971144014c1f135c05e"
server_key_iv: "30b04d0fe094e28f00ca764199bfe510"
server_key_version: "encryptionkey@0"
I20260926 12:37:43.415133 31666 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.001s
I20260926 12:37:43.417596 31681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:43.418876 31666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:37:43.418979 31666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "575238d3efc442b7a7a76228530abb9d"
format_stamp: "Formatted at 2026-09-26 12:37:43 on dist-test-slave-t7tk"
server_key: "0dc393b447939971144014c1f135c05e"
server_key_iv: "30b04d0fe094e28f00ca764199bfe510"
server_key_version: "encryptionkey@0"
I20260926 12:37:43.419090 31666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:43.439937 31666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:43.443225 31666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:43.443408 31666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:43.450989 31666 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39119
I20260926 12:37:43.451009 31733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39119 every 8 connection(s)
I20260926 12:37:43.451977 31666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:43.454849 31734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:43.457062 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 31666
I20260926 12:37:43.457221 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:43.457542 10632 external_mini_cluster.cc:1428] Setting key 27e9b99e6db9b35b3e6a3eebdb1fea74
I20260926 12:37:43.461218 31734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d: Bootstrap starting.
I20260926 12:37:43.463871 31734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:43.464745 31734 log.cc:824] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:43.466974 31734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d: No bootstrap required, opened a new log
I20260926 12:37:43.467130 31737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:43.459049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:43557 (local address 127.0.0.1:39119)
0926 12:37:43.459475 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:43.459494 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:43.459530 (+ 36us) server_negotiation.cc:407] Connection header received
0926 12:37:43.460194 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:43.460212 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:43.460592 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:43.460956 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:43.461399 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:43.462394 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:43.463654 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:43.464279 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:43.464440 (+ 161us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:43.465398 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:43.465403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:43.465417 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:43.465816 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:43.465872 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:43.465896 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:43.466030 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:43.466215 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 12:37:43.466356 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":117,"threads_started":1}
I20260926 12:37:43.470474 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:37:43.471226 31734 raft_consensus.cc:348] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575238d3efc442b7a7a76228530abb9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } }
I20260926 12:37:43.471481 31734 raft_consensus.cc:374] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:43.471520 31734 raft_consensus.cc:729] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575238d3efc442b7a7a76228530abb9d, State: Initialized, Role: FOLLOWER
I20260926 12:37:43.472034 31734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575238d3efc442b7a7a76228530abb9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } }
I20260926 12:37:43.472185 31734 raft_consensus.cc:388] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:43.472255 31734 raft_consensus.cc:482] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:43.472357 31734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:43.473410 31734 raft_consensus.cc:504] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575238d3efc442b7a7a76228530abb9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } }
I20260926 12:37:43.473757 31734 leader_election.cc:302] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 575238d3efc442b7a7a76228530abb9d; no voters:
I20260926 12:37:43.474045 31734 leader_election.cc:288] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:43.474145 31744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:43.474318 31744 raft_consensus.cc:686] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d [term 1 LEADER]: Becoming Leader. State: Replica: 575238d3efc442b7a7a76228530abb9d, State: Running, Role: LEADER
I20260926 12:37:43.474679 31744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575238d3efc442b7a7a76228530abb9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } }
I20260926 12:37:43.475435 31734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:43.476768 31745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "575238d3efc442b7a7a76228530abb9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575238d3efc442b7a7a76228530abb9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } } }
I20260926 12:37:43.476909 31745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:43.477139 31737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:43.472493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:49385 (local address 127.0.0.1:39119)
0926 12:37:43.472614 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:43.472618 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:43.472754 (+ 136us) server_negotiation.cc:407] Connection header received
0926 12:37:43.472881 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:43.472884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:43.472931 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:43.473039 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:43.473325 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:43.473917 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:43.474951 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:43.475201 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:43.475302 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:43.476564 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:43.476568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:43.476571 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:43.476585 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:43.476616 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:43.476630 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:43.476715 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:43.476928 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 12:37:43.476968 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:37:43.477528 31746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 575238d3efc442b7a7a76228530abb9d. Latest consensus state: current_term: 1 leader_uuid: "575238d3efc442b7a7a76228530abb9d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "575238d3efc442b7a7a76228530abb9d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39119 } } }
I20260926 12:37:43.477735 31746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:43.477669 31753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:43.481168 31753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:43.486841 31753 catalog_manager.cc:1347] Generated new cluster ID: e2e57263721743c8a54bf5ab1da8d400
I20260926 12:37:43.486924 31753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:43.520927 31753 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:43.522038 31753 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:43.537269 31753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 575238d3efc442b7a7a76228530abb9d: Generated new TSK 0
I20260926 12:37:43.538084 31753 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:37:43.551759 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 31666
W20260926 12:37:43.558597 31651 connection.cc:535] client connection to 127.0.0.1:39119 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39119): Connection reset by peer (error 104)
2026-09-26T12:37:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1389 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:43 dist-test-slave-t7tk krb5kdc[31765](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:43 dist-test-slave-t7tk krb5kdc[31765](info): set up 2 sockets
Sep 26 12:37:43 dist-test-slave-t7tk krb5kdc[31765](info): commencing operation
krb5kdc: starting...
W20260926 12:37:44.523289 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.950s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:44 dist-test-slave-t7tk krb5kdc[31765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:37:44.649163 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39633
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39633
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:44.761500 31778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:44.761801 31778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:44.761862 31778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:44.766016 31778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:44.766093 31778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:44.766119 31778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:44.766140 31778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:44.766166 31778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:44.771569 31778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39633
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31778
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:44.772794 31778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:44.773905 31778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:44.782374 31783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:44.782521 31778 server_base.cc:1034] running on GCE node
W20260926 12:37:44.782626 31784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:44.782392 31786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:44.783252 31778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:44.784422 31778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:44.785606 31778 hybrid_clock.cc:648] HybridClock initialized: now 1790426264785590 us; error 30 us; skew 500 ppm
Sep 26 12:37:44 dist-test-slave-t7tk krb5kdc[31765](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426264, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:44.788651 31778 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:37:44.789585 31778 webserver.cc:460] Webserver started at http://127.0.0.1:36455/ using document root <none> and password file <none>
I20260926 12:37:44.790140 31778 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:44.790206 31778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:44.790398 31778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:44.792697 31778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "9e5b106475f444359336f25a5dca73a1"
format_stamp: "Formatted at 2026-09-26 12:37:44 on dist-test-slave-t7tk"
server_key: "7c595b5ec997cdc815cfda343f7422ff"
server_key_iv: "54eabad7ef62813d40a6dc36595cec69"
server_key_version: "encryptionkey@0"
I20260926 12:37:44.793377 31778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "9e5b106475f444359336f25a5dca73a1"
format_stamp: "Formatted at 2026-09-26 12:37:44 on dist-test-slave-t7tk"
server_key: "7c595b5ec997cdc815cfda343f7422ff"
server_key_iv: "54eabad7ef62813d40a6dc36595cec69"
server_key_version: "encryptionkey@0"
I20260926 12:37:44.797609 31778 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:37:44.800130 31793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:44.801481 31778 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:37:44.801630 31778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "9e5b106475f444359336f25a5dca73a1"
format_stamp: "Formatted at 2026-09-26 12:37:44 on dist-test-slave-t7tk"
server_key: "7c595b5ec997cdc815cfda343f7422ff"
server_key_iv: "54eabad7ef62813d40a6dc36595cec69"
server_key_version: "encryptionkey@0"
I20260926 12:37:44.801730 31778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:44.806385 31778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:44.809597 31778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:44.809769 31778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:44.816691 31778 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39633
I20260926 12:37:44.816783 31845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39633 every 8 connection(s)
I20260926 12:37:44.817684 31778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:44.820494 31846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:44.825619 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 31778
I20260926 12:37:44.825758 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:44.826007 10632 external_mini_cluster.cc:1428] Setting key 56737174e3bde7e23fe5f01e155e08d5
I20260926 12:37:44.826164 31846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1: Bootstrap starting.
I20260926 12:37:44.828591 31846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:44.829463 31846 log.cc:824] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:44.831095 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:44.827251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:44479 (local address 127.0.0.1:39633)
0926 12:37:44.827729 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:44.827743 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:44.827779 (+ 36us) server_negotiation.cc:407] Connection header received
0926 12:37:44.828407 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:44.828424 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:44.828715 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:44.829026 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:44.829866 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:44.829871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:44.829883 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:44.829894 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:44.829948 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:44.829967 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:44.830068 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:44.830208 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 12:37:44.830377 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":113,"threads_started":1}
I20260926 12:37:44.831519 31846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1: No bootstrap required, opened a new log
I20260926 12:37:44.834112 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:37:44.835621 31846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5b106475f444359336f25a5dca73a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39633 } }
I20260926 12:37:44.835851 31846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:44.835911 31846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e5b106475f444359336f25a5dca73a1, State: Initialized, Role: FOLLOWER
I20260926 12:37:44.836375 31846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5b106475f444359336f25a5dca73a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39633 } }
I20260926 12:37:44.836561 31846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:44.836638 31846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:44.836737 31846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:44.837797 31846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5b106475f444359336f25a5dca73a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39633 } }
I20260926 12:37:44.837889 31849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:44.836188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:34165 (local address 127.0.0.1:39633)
0926 12:37:44.836316 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:44.836321 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:44.836526 (+ 205us) server_negotiation.cc:407] Connection header received
0926 12:37:44.836612 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:44.836615 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:44.836655 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:44.836732 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:44.837500 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:44.837504 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:44.837506 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:44.837518 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:44.837549 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:44.837562 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:44.837652 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:44.837707 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 12:37:44.837747 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 12:37:44.838186 31846 leader_election.cc:302] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e5b106475f444359336f25a5dca73a1; no voters:
I20260926 12:37:44.838498 31846 leader_election.cc:288] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:44.838577 31856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:44.838773 31856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1 [term 1 LEADER]: Becoming Leader. State: Replica: 9e5b106475f444359336f25a5dca73a1, State: Running, Role: LEADER
I20260926 12:37:44.839125 31856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5b106475f444359336f25a5dca73a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39633 } }
I20260926 12:37:44.839845 31846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:44.840688 31858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e5b106475f444359336f25a5dca73a1. Latest consensus state: current_term: 1 leader_uuid: "9e5b106475f444359336f25a5dca73a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5b106475f444359336f25a5dca73a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39633 } } }
I20260926 12:37:44.840873 31858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:44.841171 31858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e5b106475f444359336f25a5dca73a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e5b106475f444359336f25a5dca73a1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39633 } } }
I20260926 12:37:44.841303 31858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:44.841317 31865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:44.844395 31865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:44.849669 31865 catalog_manager.cc:1347] Generated new cluster ID: 745495add2f0452ea4e85fd160f84460
I20260926 12:37:44.849810 31865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:44.866909 31865 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:44.867276 31865 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:44.875095 31865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e5b106475f444359336f25a5dca73a1: Generated new TSK 0
I20260926 12:37:44.875837 31865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:37:44.886855 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 31778
2026-09-26T12:37:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:44 dist-test-slave-t7tk krb5kdc[31877](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:44 dist-test-slave-t7tk krb5kdc[31877](info): set up 2 sockets
Sep 26 12:37:44 dist-test-slave-t7tk krb5kdc[31877](info): commencing operation
krb5kdc: starting...
W20260926 12:37:45.846624 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.941s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:45 dist-test-slave-t7tk krb5kdc[31877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:37:45.968257 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44465
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33655
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:46.077975 31890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:46.078258 31890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:46.078301 31890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:46.082316 31890 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:46.082386 31890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:46.082404 31890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:46.082422 31890 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:46.082435 31890 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:46.087883 31890 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33655
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44465
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:46.089030 31890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:46.090128 31890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:46.099383 31895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:46.099385 31898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:46.099493 31896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:46.099776 31890 server_base.cc:1034] running on GCE node
I20260926 12:37:46.100396 31890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:46.101823 31890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:46.102999 31890 hybrid_clock.cc:648] HybridClock initialized: now 1790426266102982 us; error 34 us; skew 500 ppm
Sep 26 12:37:46 dist-test-slave-t7tk krb5kdc[31877](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426266, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:46.105962 31890 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:37:46.106964 31890 webserver.cc:460] Webserver started at http://127.0.0.1:34227/ using document root <none> and password file <none>
I20260926 12:37:46.107522 31890 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:46.107590 31890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:46.107784 31890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:46.109932 31890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "0b5e0287237143ee8f431f341c5e94ce"
format_stamp: "Formatted at 2026-09-26 12:37:46 on dist-test-slave-t7tk"
server_key: "b54244d96aa1255e04e3040bbd46adad"
server_key_iv: "7aa2284315b7a0bdc243743cc77397eb"
server_key_version: "encryptionkey@0"
I20260926 12:37:46.110482 31890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "0b5e0287237143ee8f431f341c5e94ce"
format_stamp: "Formatted at 2026-09-26 12:37:46 on dist-test-slave-t7tk"
server_key: "b54244d96aa1255e04e3040bbd46adad"
server_key_iv: "7aa2284315b7a0bdc243743cc77397eb"
server_key_version: "encryptionkey@0"
I20260926 12:37:46.114259 31890 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:37:46.116916 31905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:46.118302 31890 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:37:46.118435 31890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "0b5e0287237143ee8f431f341c5e94ce"
format_stamp: "Formatted at 2026-09-26 12:37:46 on dist-test-slave-t7tk"
server_key: "b54244d96aa1255e04e3040bbd46adad"
server_key_iv: "7aa2284315b7a0bdc243743cc77397eb"
server_key_version: "encryptionkey@0"
I20260926 12:37:46.118552 31890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:46.136621 31890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:46.139971 31890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:46.140156 31890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:46.147173 31890 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44465
I20260926 12:37:46.147322 31957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44465 every 8 connection(s)
I20260926 12:37:46.148430 31890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:46.151469 31958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:46.154808 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 31890
I20260926 12:37:46.154933 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:46.155184 10632 external_mini_cluster.cc:1428] Setting key 9f686ef3408b0f742ec92e21976c8787
I20260926 12:37:46.157370 31958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce: Bootstrap starting.
I20260926 12:37:46.159870 31958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:46.160609 31958 log.cc:824] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:46.162909 31958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce: No bootstrap required, opened a new log
Sep 26 12:37:46 dist-test-slave-t7tk krb5kdc[31877](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426265, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 12:37:46.166431 31958 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5e0287237143ee8f431f341c5e94ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44465 } }
I20260926 12:37:46.166664 31958 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:46.166711 31958 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b5e0287237143ee8f431f341c5e94ce, State: Initialized, Role: FOLLOWER
I20260926 12:37:46.167164 31958 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5e0287237143ee8f431f341c5e94ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44465 } }
I20260926 12:37:46.167297 31958 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:46.167358 31958 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:46.167456 31958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:46.168519 31958 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5e0287237143ee8f431f341c5e94ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44465 } }
I20260926 12:37:46.168853 31958 leader_election.cc:302] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b5e0287237143ee8f431f341c5e94ce; no voters:
I20260926 12:37:46.169205 31958 leader_election.cc:288] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:46.169282 31963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:46.169550 31963 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce [term 1 LEADER]: Becoming Leader. State: Replica: 0b5e0287237143ee8f431f341c5e94ce, State: Running, Role: LEADER
I20260926 12:37:46.169890 31963 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5e0287237143ee8f431f341c5e94ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44465 } }
I20260926 12:37:46.170490 31958 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:46.170874 31961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:46.156670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:51625 (local address 127.0.0.1:44465)
0926 12:37:46.157079 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:46.157117 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:46.157161 (+ 44us) server_negotiation.cc:407] Connection header received
0926 12:37:46.157833 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:46.157852 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:46.158232 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:46.158516 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:46.159431 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:46.160513 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:46.161601 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:46.162109 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:46.162240 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:46.165080 (+ 2840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:46.165085 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:46.165122 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:46.165158 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:46.167308 (+ 2150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:46.167923 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:46.167928 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:46.167933 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:46.168048 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:46.168394 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:46.168398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:46.168400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:46.168854 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:46.169071 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:46.169533 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 12:37:46.169835 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":103,"threads_started":1}
I20260926 12:37:46.171797 31964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b5e0287237143ee8f431f341c5e94ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5e0287237143ee8f431f341c5e94ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44465 } } }
I20260926 12:37:46.172159 31964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:46.172057 31965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b5e0287237143ee8f431f341c5e94ce. Latest consensus state: current_term: 1 leader_uuid: "0b5e0287237143ee8f431f341c5e94ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b5e0287237143ee8f431f341c5e94ce" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44465 } } }
I20260926 12:37:46.172462 31965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:46.172684 31977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:46.174134 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:37:46.176276 31977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:46.181713 31977 catalog_manager.cc:1347] Generated new cluster ID: 7750131d37984ed2a08c41cd0e93a468
I20260926 12:37:46.181779 31977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:46.186213 31961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:46.176625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:36387 (local address 127.0.0.1:44465)
0926 12:37:46.176745 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:46.176749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:46.177442 (+ 693us) server_negotiation.cc:407] Connection header received
0926 12:37:46.177556 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:46.177560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:46.177612 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:46.177693 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:46.178629 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:46.179367 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:46.180486 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:46.180880 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:46.180957 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:46.182821 (+ 1864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:46.182825 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:46.182828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:46.182853 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:46.184402 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:46.184911 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:46.184914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:46.184916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:46.184986 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:46.185334 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:46.185337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:46.185339 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:46.185529 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:46.185683 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:46.185969 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 12:37:46.186061 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:37:46.195801 31977 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:46.196938 31977 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:46.203986 31977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b5e0287237143ee8f431f341c5e94ce: Generated new TSK 0
I20260926 12:37:46.204815 31977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:37:46.211401 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 31890
W20260926 12:37:46.216121 31875 connection.cc:535] client connection to 127.0.0.1:44465 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44465): Connection reset by peer (error 104)
2026-09-26T12:37:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:46 dist-test-slave-t7tk krb5kdc[31989](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:46 dist-test-slave-t7tk krb5kdc[31989](info): set up 2 sockets
Sep 26 12:37:46 dist-test-slave-t7tk krb5kdc[31989](info): commencing operation
krb5kdc: starting...
W20260926 12:37:47.186666 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.956s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:47 dist-test-slave-t7tk krb5kdc[31989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426267, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:37:47.304270 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36565
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:47.412264 32002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:47.412537 32002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:47.412586 32002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:47.416536 32002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:47.416695 32002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:47.416746 32002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:47.416769 32002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:47.416783 32002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:47.422495 32002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36565
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:47.423522 32002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:47.424480 32002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:37:47.432832 32002 server_base.cc:1034] running on GCE node
W20260926 12:37:47.432865 32007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:47.432889 32008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:47.432902 32010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:47.433683 32002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:47.434769 32002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:47.435938 32002 hybrid_clock.cc:648] HybridClock initialized: now 1790426267435909 us; error 42 us; skew 500 ppm
Sep 26 12:37:47 dist-test-slave-t7tk krb5kdc[31989](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:47.438700 32002 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:37:47.439600 32002 webserver.cc:460] Webserver started at http://127.0.0.1:37309/ using document root <none> and password file <none>
I20260926 12:37:47.440126 32002 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:47.440187 32002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:47.440373 32002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:47.442451 32002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "6cb1cbd303574e8fa4f68bf7c2fd2954"
format_stamp: "Formatted at 2026-09-26 12:37:47 on dist-test-slave-t7tk"
server_key: "28ca4d5a60ccfc391980dcb37e78d346"
server_key_iv: "c71dc60d8d296940516d47b246433bf3"
server_key_version: "encryptionkey@0"
I20260926 12:37:47.442998 32002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "6cb1cbd303574e8fa4f68bf7c2fd2954"
format_stamp: "Formatted at 2026-09-26 12:37:47 on dist-test-slave-t7tk"
server_key: "28ca4d5a60ccfc391980dcb37e78d346"
server_key_iv: "c71dc60d8d296940516d47b246433bf3"
server_key_version: "encryptionkey@0"
I20260926 12:37:47.446702 32002 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260926 12:37:47.449373 32017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:47.450714 32002 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:37:47.450824 32002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "6cb1cbd303574e8fa4f68bf7c2fd2954"
format_stamp: "Formatted at 2026-09-26 12:37:47 on dist-test-slave-t7tk"
server_key: "28ca4d5a60ccfc391980dcb37e78d346"
server_key_iv: "c71dc60d8d296940516d47b246433bf3"
server_key_version: "encryptionkey@0"
I20260926 12:37:47.450934 32002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:47.464089 32002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:47.467281 32002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:47.467461 32002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:47.474398 32002 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36565
I20260926 12:37:47.474418 32069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36565 every 8 connection(s)
I20260926 12:37:47.475343 32002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:47.478139 32070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:47.480098 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 32002
I20260926 12:37:47.480212 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:47.480463 10632 external_mini_cluster.cc:1428] Setting key 02e067704ae6d61333aaf6995452f96c
I20260926 12:37:47.484042 32070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954: Bootstrap starting.
I20260926 12:37:47.486327 32070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:47.487133 32070 log.cc:824] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:47.489035 32070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954: No bootstrap required, opened a new log
I20260926 12:37:47.489459 32073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:47.481867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:53909 (local address 127.0.0.1:36565)
0926 12:37:47.482316 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:47.482331 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:47.482365 (+ 34us) server_negotiation.cc:407] Connection header received
0926 12:37:47.483188 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:47.483208 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:47.483573 (+ 365us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:47.483893 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:47.484336 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:47.485372 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:47.486295 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:47.486797 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:47.486942 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:47.487774 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:47.487779 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:47.487793 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:47.488182 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:47.488235 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:47.488258 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:47.488366 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:47.488554 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 12:37:47.488683 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":135,"threads_started":1}
I20260926 12:37:47.492182 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:37:47.492717 32070 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb1cbd303574e8fa4f68bf7c2fd2954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36565 } }
I20260926 12:37:47.492941 32070 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:47.492986 32070 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb1cbd303574e8fa4f68bf7c2fd2954, State: Initialized, Role: FOLLOWER
I20260926 12:37:47.493433 32070 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb1cbd303574e8fa4f68bf7c2fd2954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36565 } }
I20260926 12:37:47.493593 32070 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:47.493660 32070 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:47.493757 32070 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:47.494750 32070 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb1cbd303574e8fa4f68bf7c2fd2954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36565 } }
I20260926 12:37:47.495082 32070 leader_election.cc:302] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cb1cbd303574e8fa4f68bf7c2fd2954; no voters:
I20260926 12:37:47.495357 32070 leader_election.cc:288] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:47.495457 32080 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:47.495674 32080 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954 [term 1 LEADER]: Becoming Leader. State: Replica: 6cb1cbd303574e8fa4f68bf7c2fd2954, State: Running, Role: LEADER
I20260926 12:37:47.496042 32080 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb1cbd303574e8fa4f68bf7c2fd2954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36565 } }
I20260926 12:37:47.496665 32070 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:47.497206 32080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cb1cbd303574e8fa4f68bf7c2fd2954. Latest consensus state: current_term: 1 leader_uuid: "6cb1cbd303574e8fa4f68bf7c2fd2954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb1cbd303574e8fa4f68bf7c2fd2954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36565 } } }
I20260926 12:37:47.497334 32080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:47.497598 32081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cb1cbd303574e8fa4f68bf7c2fd2954" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cb1cbd303574e8fa4f68bf7c2fd2954" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36565 } } }
I20260926 12:37:47.497749 32081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:47.499032 32073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:47.494232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:49749 (local address 127.0.0.1:36565)
0926 12:37:47.494391 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:47.494395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:47.494599 (+ 204us) server_negotiation.cc:407] Connection header received
0926 12:37:47.494715 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:47.494718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:47.494760 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:47.494819 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:47.495074 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:47.495766 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:47.496853 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:47.497216 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:47.497315 (+ 99us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:47.498280 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:47.498284 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:47.498287 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:47.498300 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:47.498329 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:47.498342 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:47.498430 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:47.498871 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 12:37:47.498906 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260926 12:37:47.501945 32095 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:37:47.502014 32095 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:37:47.502137 32085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:47.503007 32085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:47.508340 32085 catalog_manager.cc:1347] Generated new cluster ID: 857174087d0643608301e93ef0b574c2
I20260926 12:37:47.508410 32085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:47.532969 32085 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:47.533951 32085 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:47.538542 32085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cb1cbd303574e8fa4f68bf7c2fd2954: Generated new TSK 0
I20260926 12:37:47.539254 32085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:37:47.543004 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 32002
W20260926 12:37:47.548048 31987 connection.cc:535] client connection to 127.0.0.1:36565 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36565): Connection reset by peer (error 104)
2026-09-26T12:37:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:47 dist-test-slave-t7tk krb5kdc[32101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:47 dist-test-slave-t7tk krb5kdc[32101](info): set up 2 sockets
Sep 26 12:37:47 dist-test-slave-t7tk krb5kdc[32101](info): commencing operation
krb5kdc: starting...
W20260926 12:37:48.499022 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:48 dist-test-slave-t7tk krb5kdc[32101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426268, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 12:37:48.616752 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44477
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:48.726168 32114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:48.726424 32114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:48.726464 32114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:48.730338 32114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:48.730391 32114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:48.730408 32114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:48.730424 32114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:48.730437 32114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:48.735553 32114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44477
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32114
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:48.736498 32114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:48.737437 32114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:37:48.745568 32114 server_base.cc:1034] running on GCE node
W20260926 12:37:48.745651 32122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:48.745508 32119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:48.745504 32120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:48.746265 32114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:48.747328 32114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:48.748478 32114 hybrid_clock.cc:648] HybridClock initialized: now 1790426268748433 us; error 61 us; skew 500 ppm
Sep 26 12:37:48 dist-test-slave-t7tk krb5kdc[32101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:48.757542 32114 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 12:37:48.758495 32114 webserver.cc:460] Webserver started at http://127.0.0.1:46447/ using document root <none> and password file <none>
I20260926 12:37:48.759069 32114 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:48.759140 32114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:48.759344 32114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:48.761439 32114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "eb67cfdbead4499eb640b1ecaad9600b"
format_stamp: "Formatted at 2026-09-26 12:37:48 on dist-test-slave-t7tk"
server_key: "f01a1f2b3095ffec7590a537b3a88207"
server_key_iv: "78973ea2d1b7b9bb024494aa260c8709"
server_key_version: "encryptionkey@0"
I20260926 12:37:48.761981 32114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "eb67cfdbead4499eb640b1ecaad9600b"
format_stamp: "Formatted at 2026-09-26 12:37:48 on dist-test-slave-t7tk"
server_key: "f01a1f2b3095ffec7590a537b3a88207"
server_key_iv: "78973ea2d1b7b9bb024494aa260c8709"
server_key_version: "encryptionkey@0"
I20260926 12:37:48.765609 32114 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260926 12:37:48.767887 32129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:48.769078 32114 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:37:48.769232 32114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "eb67cfdbead4499eb640b1ecaad9600b"
format_stamp: "Formatted at 2026-09-26 12:37:48 on dist-test-slave-t7tk"
server_key: "f01a1f2b3095ffec7590a537b3a88207"
server_key_iv: "78973ea2d1b7b9bb024494aa260c8709"
server_key_version: "encryptionkey@0"
I20260926 12:37:48.769323 32114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:48.773667 32114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:48.776557 32114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:48.776715 32114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:48.783459 32114 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44477
I20260926 12:37:48.783480 32181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44477 every 8 connection(s)
I20260926 12:37:48.784344 32114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:48.787223 32182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:48.792599 32182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b: Bootstrap starting.
I20260926 12:37:48.792856 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 32114
I20260926 12:37:48.792955 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:48.793205 10632 external_mini_cluster.cc:1428] Setting key da3035011abfd5c65fba8f1d9982a82d
I20260926 12:37:48.794974 32182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:48.795655 32182 log.cc:824] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:48.797755 32182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b: No bootstrap required, opened a new log
I20260926 12:37:48.797973 32185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:48.794455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:32811 (local address 127.0.0.1:44477)
0926 12:37:48.794848 (+ 393us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:48.794861 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:48.794895 (+ 34us) server_negotiation.cc:407] Connection header received
0926 12:37:48.795519 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:48.795536 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:48.795826 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:48.796157 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:48.796820 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:48.796824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:48.796835 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:48.796843 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:48.796887 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:48.796904 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:48.796980 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:48.797116 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 12:37:48.797271 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":107,"threads_started":1}
I20260926 12:37:48.800459 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:37:48.801349 32182 raft_consensus.cc:348] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb67cfdbead4499eb640b1ecaad9600b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44477 } }
I20260926 12:37:48.801631 32182 raft_consensus.cc:374] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:48.801736 32182 raft_consensus.cc:729] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb67cfdbead4499eb640b1ecaad9600b, State: Initialized, Role: FOLLOWER
I20260926 12:37:48.802165 32182 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb67cfdbead4499eb640b1ecaad9600b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44477 } }
I20260926 12:37:48.802325 32182 raft_consensus.cc:388] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:48.802394 32182 raft_consensus.cc:482] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:48.802489 32182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:48.803361 32182 raft_consensus.cc:504] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb67cfdbead4499eb640b1ecaad9600b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44477 } }
I20260926 12:37:48.803725 32182 leader_election.cc:302] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb67cfdbead4499eb640b1ecaad9600b; no voters:
I20260926 12:37:48.803825 32185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:48.802289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.176:47069 (local address 127.0.0.1:44477)
0926 12:37:48.802428 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:48.802432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:48.802607 (+ 175us) server_negotiation.cc:407] Connection header received
0926 12:37:48.802693 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:48.802696 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:48.802727 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:48.802786 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:48.803417 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:48.803420 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:48.803423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:48.803434 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:48.803460 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:48.803471 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:48.803545 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:48.803647 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 12:37:48.803679 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 12:37:48.804061 32182 leader_election.cc:288] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:48.804144 32192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:48.804378 32192 raft_consensus.cc:686] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b [term 1 LEADER]: Becoming Leader. State: Replica: eb67cfdbead4499eb640b1ecaad9600b, State: Running, Role: LEADER
I20260926 12:37:48.804733 32192 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb67cfdbead4499eb640b1ecaad9600b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44477 } }
I20260926 12:37:48.805359 32182 sys_catalog.cc:564] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:48.805855 32192 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb67cfdbead4499eb640b1ecaad9600b. Latest consensus state: current_term: 1 leader_uuid: "eb67cfdbead4499eb640b1ecaad9600b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb67cfdbead4499eb640b1ecaad9600b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44477 } } }
I20260926 12:37:48.805963 32192 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:48.806293 32193 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb67cfdbead4499eb640b1ecaad9600b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb67cfdbead4499eb640b1ecaad9600b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44477 } } }
I20260926 12:37:48.806392 32197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:48.806427 32193 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:48.809573 32197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:48.814827 32197 catalog_manager.cc:1347] Generated new cluster ID: 3ff8af784f1c4a9d897b7d07959dc126
I20260926 12:37:48.814883 32197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:48.828018 32197 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:48.828320 32197 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:48.832852 32197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eb67cfdbead4499eb640b1ecaad9600b: Generated new TSK 0
I20260926 12:37:48.833722 32197 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:37:48.839485 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 32114
2026-09-26T12:37:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1295 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:48 dist-test-slave-t7tk krb5kdc[32213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:48 dist-test-slave-t7tk krb5kdc[32213](info): set up 2 sockets
Sep 26 12:37:48 dist-test-slave-t7tk krb5kdc[32213](info): commencing operation
krb5kdc: starting...
W20260926 12:37:49.798517 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:49 dist-test-slave-t7tk krb5kdc[32213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:37:49.916160 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:38571
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:34109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:38571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:50.025600 32226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:50.025897 32226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:50.025941 32226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:50.030036 32226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:50.030117 32226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:50.030136 32226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:50.030151 32226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:50.030165 32226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:50.036005 32226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:34109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:38571
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:38571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:50.037259 32226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:50.038297 32226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:50.047030 32232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:50.047034 32231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:50.047122 32226 server_base.cc:1034] running on GCE node
W20260926 12:37:50.047034 32234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:50.047892 32226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:50.048995 32226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:50.050170 32226 hybrid_clock.cc:648] HybridClock initialized: now 1790426270050151 us; error 30 us; skew 500 ppm
Sep 26 12:37:50 dist-test-slave-t7tk krb5kdc[32213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426270, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:50.052729 32226 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:37:50.053627 32226 webserver.cc:460] Webserver started at http://127.10.98.62:39309/ using document root <none> and password file <none>
I20260926 12:37:50.054138 32226 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:50.054193 32226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:50.054352 32226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:50.056313 32226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "49340d56e33141b88e7745286dff54ad"
format_stamp: "Formatted at 2026-09-26 12:37:50 on dist-test-slave-t7tk"
server_key: "a355352a2fc6b2e134696c1bce1bcd43"
server_key_iv: "1e96bdc9c544bba31a5456739d1fe8fa"
server_key_version: "encryptionkey@0"
I20260926 12:37:50.056807 32226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "49340d56e33141b88e7745286dff54ad"
format_stamp: "Formatted at 2026-09-26 12:37:50 on dist-test-slave-t7tk"
server_key: "a355352a2fc6b2e134696c1bce1bcd43"
server_key_iv: "1e96bdc9c544bba31a5456739d1fe8fa"
server_key_version: "encryptionkey@0"
I20260926 12:37:50.060345 32226 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260926 12:37:50.062736 32241 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:50.063927 32226 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:37:50.064031 32226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "49340d56e33141b88e7745286dff54ad"
format_stamp: "Formatted at 2026-09-26 12:37:50 on dist-test-slave-t7tk"
server_key: "a355352a2fc6b2e134696c1bce1bcd43"
server_key_iv: "1e96bdc9c544bba31a5456739d1fe8fa"
server_key_version: "encryptionkey@0"
I20260926 12:37:50.064167 32226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:50.081419 32226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:50.084506 32226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:50.084677 32226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:50.091331 32226 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:38571
I20260926 12:37:50.091339 32293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:38571 every 8 connection(s)
I20260926 12:37:50.092278 32226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:50.095129 32294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:50.100687 32294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad: Bootstrap starting.
I20260926 12:37:50.102300 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 32226
I20260926 12:37:50.102443 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:50.102725 10632 external_mini_cluster.cc:1428] Setting key 897f1f0005ec98cb1e434631e431e769
I20260926 12:37:50.103452 32294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:50.104327 32294 log.cc:824] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:50.106307 32294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad: No bootstrap required, opened a new log
I20260926 12:37:50.109750 32294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49340d56e33141b88e7745286dff54ad" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38571 } }
I20260926 12:37:50.109998 32294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:50.110057 32294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49340d56e33141b88e7745286dff54ad, State: Initialized, Role: FOLLOWER
I20260926 12:37:50.110498 32294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49340d56e33141b88e7745286dff54ad" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38571 } }
I20260926 12:37:50.110637 32294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:50.110703 32294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:50.110800 32294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad [term 0 FOLLOWER]: Advancing to term 1
Sep 26 12:37:50 dist-test-slave-t7tk krb5kdc[32213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426269, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:37:50.111707 32294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49340d56e33141b88e7745286dff54ad" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38571 } }
I20260926 12:37:50.112044 32294 leader_election.cc:302] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49340d56e33141b88e7745286dff54ad; no voters:
I20260926 12:37:50.112385 32294 leader_election.cc:288] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:50.112466 32299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:50.112664 32299 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad [term 1 LEADER]: Becoming Leader. State: Replica: 49340d56e33141b88e7745286dff54ad, State: Running, Role: LEADER
I20260926 12:37:50.112957 32299 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49340d56e33141b88e7745286dff54ad" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38571 } }
I20260926 12:37:50.113726 32294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:50.114821 32300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49340d56e33141b88e7745286dff54ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49340d56e33141b88e7745286dff54ad" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38571 } } }
I20260926 12:37:50.114830 32301 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49340d56e33141b88e7745286dff54ad. Latest consensus state: current_term: 1 leader_uuid: "49340d56e33141b88e7745286dff54ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49340d56e33141b88e7745286dff54ad" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 38571 } } }
I20260926 12:37:50.115026 32300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:50.115032 32301 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:50.115468 32309 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:50.117400 32297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:50.103839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56345 (local address 127.10.98.62:38571)
0926 12:37:50.104298 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:50.104313 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:50.104348 (+ 35us) server_negotiation.cc:407] Connection header received
0926 12:37:50.105213 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:50.105233 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:50.105590 (+ 357us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:50.105906 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:50.106684 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:50.107677 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:50.108538 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:50.109020 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:50.109207 (+ 187us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:50.111714 (+ 2507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:50.111720 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:50.111733 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:50.111760 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:50.113586 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:50.114171 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:50.114177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:50.114183 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:50.114319 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:50.114729 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:50.114732 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:50.114734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:50.115258 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:50.116021 (+ 763us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:50.116225 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 12:37:50.116451 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":146,"threads_started":1}
I20260926 12:37:50.119684 32309 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:50.120293 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:37:50.125562 32309 catalog_manager.cc:1347] Generated new cluster ID: 37d8dabbccd84cf0831a812a61dddd4e
I20260926 12:37:50.125630 32309 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:50.130836 32297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:50.122315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37317 (local address 127.10.98.62:38571)
0926 12:37:50.122413 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:50.122417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:50.122715 (+ 298us) server_negotiation.cc:407] Connection header received
0926 12:37:50.122830 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:50.122833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:50.122877 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:50.122963 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:50.123833 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:50.124420 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:50.125360 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:50.125632 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:50.125723 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:50.127434 (+ 1711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:50.127438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:50.127441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:50.127468 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:50.129057 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:50.129650 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:50.129654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:50.129656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:50.129745 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:50.130047 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:50.130050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:50.130051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:50.130233 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:50.130353 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:50.130600 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 12:37:50.130687 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 12:37:50.144223 32309 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:50.145254 32309 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:50.151322 32309 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49340d56e33141b88e7745286dff54ad: Generated new TSK 0
I20260926 12:37:50.152045 32309 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:37:50.157650 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 32226
W20260926 12:37:50.163187 32211 connection.cc:535] client connection to 127.10.98.62:38571 recv error: Network error: failed to read from TLS socket (remote: 127.10.98.62:38571): Connection reset by peer (error 104)
2026-09-26T12:37:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1319 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:50 dist-test-slave-t7tk krb5kdc[32325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:50 dist-test-slave-t7tk krb5kdc[32325](info): set up 2 sockets
Sep 26 12:37:50 dist-test-slave-t7tk krb5kdc[32325](info): commencing operation
krb5kdc: starting...
W20260926 12:37:51.110895 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:51 dist-test-slave-t7tk krb5kdc[32325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426271, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:37:51.230176 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:33163
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:34779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:33163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:51.338398 32338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:51.338665 32338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:51.338721 32338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:51.342705 32338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:51.342767 32338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:51.342793 32338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:51.342814 32338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:51.342834 32338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:51.348110 32338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:34779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:33163
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:33163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:51.349216 32338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:51.350257 32338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:37:51.358737 32338 server_base.cc:1034] running on GCE node
W20260926 12:37:51.358677 32343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:51.358752 32344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:51.358677 32346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:51.359453 32338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:51.360531 32338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:51.361743 32338 hybrid_clock.cc:648] HybridClock initialized: now 1790426271361715 us; error 39 us; skew 500 ppm
Sep 26 12:37:51 dist-test-slave-t7tk krb5kdc[32325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426271, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:51.364399 32338 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:37:51.365324 32338 webserver.cc:460] Webserver started at http://127.10.98.62:33233/ using document root <none> and password file <none>
I20260926 12:37:51.365849 32338 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:51.365911 32338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:51.366101 32338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:51.368144 32338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "f521f516dbc045b3a9be19d96e5c01f5"
format_stamp: "Formatted at 2026-09-26 12:37:51 on dist-test-slave-t7tk"
server_key: "d33e3c3e9c1151533ebe7ac06982d6ab"
server_key_iv: "13220087ebbb28657961c62c4d20259c"
server_key_version: "encryptionkey@0"
I20260926 12:37:51.368672 32338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "f521f516dbc045b3a9be19d96e5c01f5"
format_stamp: "Formatted at 2026-09-26 12:37:51 on dist-test-slave-t7tk"
server_key: "d33e3c3e9c1151533ebe7ac06982d6ab"
server_key_iv: "13220087ebbb28657961c62c4d20259c"
server_key_version: "encryptionkey@0"
I20260926 12:37:51.372339 32338 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260926 12:37:51.374836 32353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:51.376142 32338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260926 12:37:51.376248 32338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "f521f516dbc045b3a9be19d96e5c01f5"
format_stamp: "Formatted at 2026-09-26 12:37:51 on dist-test-slave-t7tk"
server_key: "d33e3c3e9c1151533ebe7ac06982d6ab"
server_key_iv: "13220087ebbb28657961c62c4d20259c"
server_key_version: "encryptionkey@0"
I20260926 12:37:51.376350 32338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:51.401450 32338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:51.404661 32338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:51.404838 32338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:51.411798 32338 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:33163
I20260926 12:37:51.411828 32405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:33163 every 8 connection(s)
I20260926 12:37:51.412748 32338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:51.415555 32406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:51.416314 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 32338
I20260926 12:37:51.416430 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:51.416738 10632 external_mini_cluster.cc:1428] Setting key f9141614b63b7b79149450ea43a8fc81
I20260926 12:37:51.422205 32406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5: Bootstrap starting.
I20260926 12:37:51.424775 32406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:51.425662 32406 log.cc:824] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:51.426771 32409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:51.418123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42745 (local address 127.10.98.62:33163)
0926 12:37:51.418532 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:51.418551 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:51.418594 (+ 43us) server_negotiation.cc:407] Connection header received
0926 12:37:51.419479 (+ 885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:51.419515 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:51.419964 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:51.420353 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:51.420764 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:51.422256 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:51.423170 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:51.423869 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:51.424011 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:51.424855 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:51.424861 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:51.424877 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:51.425399 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:51.425466 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:51.425497 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:51.425621 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:51.425800 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 12:37:51.425931 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":108,"threads_started":1}
I20260926 12:37:51.427819 32406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5: No bootstrap required, opened a new log
I20260926 12:37:51.429714 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:37:51.431784 32406 raft_consensus.cc:348] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f521f516dbc045b3a9be19d96e5c01f5" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 33163 } }
I20260926 12:37:51.432065 32406 raft_consensus.cc:374] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:51.432142 32406 raft_consensus.cc:729] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f521f516dbc045b3a9be19d96e5c01f5, State: Initialized, Role: FOLLOWER
I20260926 12:37:51.432595 32406 consensus_queue.cc:260] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f521f516dbc045b3a9be19d96e5c01f5" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 33163 } }
I20260926 12:37:51.432729 32406 raft_consensus.cc:388] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:51.432792 32406 raft_consensus.cc:482] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:51.432888 32406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:51.433833 32406 raft_consensus.cc:504] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f521f516dbc045b3a9be19d96e5c01f5" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 33163 } }
I20260926 12:37:51.434159 32406 leader_election.cc:302] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f521f516dbc045b3a9be19d96e5c01f5; no voters:
I20260926 12:37:51.434445 32406 leader_election.cc:288] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:51.434528 32416 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:51.435031 32416 raft_consensus.cc:686] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5 [term 1 LEADER]: Becoming Leader. State: Replica: f521f516dbc045b3a9be19d96e5c01f5, State: Running, Role: LEADER
I20260926 12:37:51.435400 32416 consensus_queue.cc:237] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f521f516dbc045b3a9be19d96e5c01f5" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 33163 } }
I20260926 12:37:51.435632 32409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:51.431696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59195 (local address 127.10.98.62:33163)
0926 12:37:51.431807 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:51.431811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:51.431965 (+ 154us) server_negotiation.cc:407] Connection header received
0926 12:37:51.432094 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:51.432097 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:51.432148 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:51.432213 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:51.432475 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:51.433078 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:51.433890 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:51.434192 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:51.434280 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:51.435159 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:51.435163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:51.435166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:51.435178 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:51.435206 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:51.435218 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:51.435268 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:51.435445 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 12:37:51.435478 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 12:37:51.435818 32406 sys_catalog.cc:564] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:51.437426 32418 sys_catalog.cc:455] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f521f516dbc045b3a9be19d96e5c01f5. Latest consensus state: current_term: 1 leader_uuid: "f521f516dbc045b3a9be19d96e5c01f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f521f516dbc045b3a9be19d96e5c01f5" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 33163 } } }
I20260926 12:37:51.437465 32417 sys_catalog.cc:455] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f521f516dbc045b3a9be19d96e5c01f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f521f516dbc045b3a9be19d96e5c01f5" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 33163 } } }
I20260926 12:37:51.437588 32418 sys_catalog.cc:458] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:51.437595 32417 sys_catalog.cc:458] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5 [sys.catalog]: This master's current role is: LEADER
W20260926 12:37:51.441000 32430 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:37:51.441074 32430 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:37:51.441179 32431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:51.441861 32431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:51.446944 32431 catalog_manager.cc:1347] Generated new cluster ID: 0cdef1e276064eff8097a3ad463c24dd
I20260926 12:37:51.447010 32431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:51.454741 32431 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:51.455618 32431 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:51.461014 32431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f521f516dbc045b3a9be19d96e5c01f5: Generated new TSK 0
I20260926 12:37:51.461704 32431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:37:51.466375 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 32338
W20260926 12:37:51.471262 32323 connection.cc:535] client connection to 127.10.98.62:33163 recv error: Network error: failed to read from TLS socket (remote: 127.10.98.62:33163): Connection reset by peer (error 104)
2026-09-26T12:37:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:51 dist-test-slave-t7tk krb5kdc[32437](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:51 dist-test-slave-t7tk krb5kdc[32437](info): set up 2 sockets
Sep 26 12:37:51 dist-test-slave-t7tk krb5kdc[32437](info): commencing operation
krb5kdc: starting...
W20260926 12:37:52.418672 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:52 dist-test-slave-t7tk krb5kdc[32437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:37:52.540410 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:35699
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:42115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:35699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:52.650486 32450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:52.650776 32450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:52.650835 32450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:52.654840 32450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:52.654903 32450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:52.654929 32450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:52.654950 32450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:52.654970 32450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:52.660301 32450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:42115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:35699
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:35699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32450
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:52.661523 32450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:52.662569 32450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:52.671391 32456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:52.671397 32458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:52.671445 32450 server_base.cc:1034] running on GCE node
W20260926 12:37:52.671397 32455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:52.672282 32450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:52.673427 32450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:52.674623 32450 hybrid_clock.cc:648] HybridClock initialized: now 1790426272674596 us; error 40 us; skew 500 ppm
Sep 26 12:37:52 dist-test-slave-t7tk krb5kdc[32437](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426272, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:52.677361 32450 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:37:52.678301 32450 webserver.cc:460] Webserver started at http://127.10.98.62:43309/ using document root <none> and password file <none>
I20260926 12:37:52.678846 32450 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:52.678911 32450 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:52.679128 32450 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:52.681197 32450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "74286913ce2441ac8eb2cd5018c86999"
format_stamp: "Formatted at 2026-09-26 12:37:52 on dist-test-slave-t7tk"
server_key: "3040c3670f71282f74ec6e63e5379ebc"
server_key_iv: "b48ccb93a749805052c12f722b67d241"
server_key_version: "encryptionkey@0"
I20260926 12:37:52.681730 32450 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "74286913ce2441ac8eb2cd5018c86999"
format_stamp: "Formatted at 2026-09-26 12:37:52 on dist-test-slave-t7tk"
server_key: "3040c3670f71282f74ec6e63e5379ebc"
server_key_iv: "b48ccb93a749805052c12f722b67d241"
server_key_version: "encryptionkey@0"
I20260926 12:37:52.685474 32450 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:37:52.688024 32465 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:52.689344 32450 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:37:52.689517 32450 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "74286913ce2441ac8eb2cd5018c86999"
format_stamp: "Formatted at 2026-09-26 12:37:52 on dist-test-slave-t7tk"
server_key: "3040c3670f71282f74ec6e63e5379ebc"
server_key_iv: "b48ccb93a749805052c12f722b67d241"
server_key_version: "encryptionkey@0"
I20260926 12:37:52.689613 32450 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:52.694100 32450 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:52.697379 32450 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:52.697546 32450 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:52.704393 32450 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:35699
I20260926 12:37:52.704496 32517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:35699 every 8 connection(s)
I20260926 12:37:52.705410 32450 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:52.706243 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 32450
I20260926 12:37:52.706352 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:52.706656 10632 external_mini_cluster.cc:1428] Setting key 1a6ae94d255b02055ec64449cf1db496
I20260926 12:37:52.708504 32518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:52.711894 32521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:52.708117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42055 (local address 127.10.98.62:35699)
0926 12:37:52.708531 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:52.708548 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:52.708594 (+ 46us) server_negotiation.cc:407] Connection header received
0926 12:37:52.709293 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:52.709312 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:52.709613 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:52.709888 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:52.710681 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:52.710685 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:52.710698 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:52.710708 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:52.710760 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:52.710778 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:52.710882 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:52.710991 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 12:37:52.711133 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":120,"threads_started":1}
I20260926 12:37:52.714509 32518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999: Bootstrap starting.
I20260926 12:37:52.714568 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:37:52.716717 32518 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:52.717519 32518 log.cc:824] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:52.717937 32521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:52.716384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40705 (local address 127.10.98.62:35699)
0926 12:37:52.716509 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:52.716513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:52.716717 (+ 204us) server_negotiation.cc:407] Connection header received
0926 12:37:52.716802 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:52.716805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:52.716841 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:52.716933 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:52.717531 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:52.717535 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:52.717538 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:52.717549 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:52.717577 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:52.717590 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:52.717673 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:52.717751 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 12:37:52.717813 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 12:37:52.719440 32518 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999: No bootstrap required, opened a new log
I20260926 12:37:52.723497 32518 raft_consensus.cc:348] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74286913ce2441ac8eb2cd5018c86999" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35699 } }
I20260926 12:37:52.723744 32518 raft_consensus.cc:374] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:52.723784 32518 raft_consensus.cc:729] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74286913ce2441ac8eb2cd5018c86999, State: Initialized, Role: FOLLOWER
I20260926 12:37:52.724203 32518 consensus_queue.cc:260] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74286913ce2441ac8eb2cd5018c86999" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35699 } }
I20260926 12:37:52.724320 32518 raft_consensus.cc:388] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:52.724363 32518 raft_consensus.cc:482] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:52.724469 32518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:52.725373 32518 raft_consensus.cc:504] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74286913ce2441ac8eb2cd5018c86999" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35699 } }
I20260926 12:37:52.725703 32518 leader_election.cc:302] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 74286913ce2441ac8eb2cd5018c86999; no voters:
I20260926 12:37:52.726001 32518 leader_election.cc:288] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:52.726212 32528 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:52.726466 32528 raft_consensus.cc:686] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999 [term 1 LEADER]: Becoming Leader. State: Replica: 74286913ce2441ac8eb2cd5018c86999, State: Running, Role: LEADER
I20260926 12:37:52.726738 32528 consensus_queue.cc:237] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74286913ce2441ac8eb2cd5018c86999" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35699 } }
I20260926 12:37:52.727140 32518 sys_catalog.cc:564] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:52.728240 32529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "74286913ce2441ac8eb2cd5018c86999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74286913ce2441ac8eb2cd5018c86999" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35699 } } }
I20260926 12:37:52.728374 32529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:52.728567 32530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 74286913ce2441ac8eb2cd5018c86999. Latest consensus state: current_term: 1 leader_uuid: "74286913ce2441ac8eb2cd5018c86999" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "74286913ce2441ac8eb2cd5018c86999" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35699 } } }
I20260926 12:37:52.728708 32530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:52.728776 32537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:52.732067 32537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:52.737460 32537 catalog_manager.cc:1347] Generated new cluster ID: bde742f569cd4c60b0af0cbf68711216
I20260926 12:37:52.737525 32537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:52.748613 32537 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:52.748891 32537 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:52.756237 32537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 74286913ce2441ac8eb2cd5018c86999: Generated new TSK 0
I20260926 12:37:52.756882 32537 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:37:52.759415 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 32450
2026-09-26T12:37:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1292 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:52 dist-test-slave-t7tk krb5kdc[32549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:52 dist-test-slave-t7tk krb5kdc[32549](info): set up 2 sockets
Sep 26 12:37:52 dist-test-slave-t7tk krb5kdc[32549](info): commencing operation
krb5kdc: starting...
W20260926 12:37:53.718619 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:53 dist-test-slave-t7tk krb5kdc[32549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:37:53.833976 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:40061
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:38399
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:40061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:53.940255 32562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:53.940536 32562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:53.940608 32562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:53.944590 32562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:53.944661 32562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:53.944689 32562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:53.944708 32562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:53.944728 32562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:53.949992 32562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:38399
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:40061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:40061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:53.951020 32562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:53.952001 32562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:53.960656 32568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:53.960708 32562 server_base.cc:1034] running on GCE node
W20260926 12:37:53.960675 32567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:53.960866 32570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:53.961419 32562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:53.962469 32562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:53.963630 32562 hybrid_clock.cc:648] HybridClock initialized: now 1790426273963614 us; error 30 us; skew 500 ppm
Sep 26 12:37:53 dist-test-slave-t7tk krb5kdc[32549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426273, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:53.972748 32562 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:37:53.973839 32562 webserver.cc:460] Webserver started at http://127.10.98.62:38887/ using document root <none> and password file <none>
I20260926 12:37:53.974416 32562 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:53.974486 32562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:53.974691 32562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:53.976769 32562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "8be234f4d94c415daf59c9d04dfd54b6"
format_stamp: "Formatted at 2026-09-26 12:37:53 on dist-test-slave-t7tk"
server_key: "67b75197df4d0239ba1e865d3fb1655d"
server_key_iv: "87cd5762d78366af11d0796df8ff623a"
server_key_version: "encryptionkey@0"
I20260926 12:37:53.977353 32562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "8be234f4d94c415daf59c9d04dfd54b6"
format_stamp: "Formatted at 2026-09-26 12:37:53 on dist-test-slave-t7tk"
server_key: "67b75197df4d0239ba1e865d3fb1655d"
server_key_iv: "87cd5762d78366af11d0796df8ff623a"
server_key_version: "encryptionkey@0"
I20260926 12:37:53.981199 32562 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260926 12:37:53.983671 32577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:53.984899 32562 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:37:53.985009 32562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "8be234f4d94c415daf59c9d04dfd54b6"
format_stamp: "Formatted at 2026-09-26 12:37:53 on dist-test-slave-t7tk"
server_key: "67b75197df4d0239ba1e865d3fb1655d"
server_key_iv: "87cd5762d78366af11d0796df8ff623a"
server_key_version: "encryptionkey@0"
I20260926 12:37:53.985177 32562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:54.015713 32562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:54.019172 32562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:54.019368 32562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:54.026260 32562 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:40061
I20260926 12:37:54.026325 32629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:40061 every 8 connection(s)
I20260926 12:37:54.027225 32562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:54.029887 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 32562
I20260926 12:37:54.030014 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:54.030153 32630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:54.030330 10632 external_mini_cluster.cc:1428] Setting key 4d9d7bbdf56728139034ac77159b4f77
I20260926 12:37:54.036549 32630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6: Bootstrap starting.
Sep 26 12:37:54 dist-test-slave-t7tk krb5kdc[32549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426273, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:37:54.038968 32630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:54.039672 32630 log.cc:824] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:54.041579 32630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6: No bootstrap required, opened a new log
I20260926 12:37:54.043377 32633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:54.031573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45313 (local address 127.10.98.62:40061)
0926 12:37:54.031963 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:54.031979 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:54.032014 (+ 35us) server_negotiation.cc:407] Connection header received
0926 12:37:54.032649 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:54.032665 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:54.032996 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:54.033342 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:54.034074 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:54.035046 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:54.035876 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:54.036342 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:54.036481 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:54.039042 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:54.039049 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:54.039067 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:54.039100 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:54.040709 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:54.041130 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:54.041137 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:54.041142 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:54.041247 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:54.041492 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:54.041496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:54.041497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:54.041916 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:54.042093 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:54.042375 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 12:37:54.042664 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":120,"threads_started":1}
I20260926 12:37:54.045349 32630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be234f4d94c415daf59c9d04dfd54b6" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40061 } }
I20260926 12:37:54.045581 32630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:54.045639 32630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8be234f4d94c415daf59c9d04dfd54b6, State: Initialized, Role: FOLLOWER
I20260926 12:37:54.046159 32630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be234f4d94c415daf59c9d04dfd54b6" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40061 } }
I20260926 12:37:54.046267 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:37:54.046307 32630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:54.046373 32630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:54.046465 32630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:54.047361 32630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be234f4d94c415daf59c9d04dfd54b6" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40061 } }
I20260926 12:37:54.047701 32630 leader_election.cc:302] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8be234f4d94c415daf59c9d04dfd54b6; no voters:
I20260926 12:37:54.047976 32630 leader_election.cc:288] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:54.048100 32639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:54.048516 32639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6 [term 1 LEADER]: Becoming Leader. State: Replica: 8be234f4d94c415daf59c9d04dfd54b6, State: Running, Role: LEADER
I20260926 12:37:54.048826 32639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be234f4d94c415daf59c9d04dfd54b6" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40061 } }
I20260926 12:37:54.049219 32630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:54.050112 32642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8be234f4d94c415daf59c9d04dfd54b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be234f4d94c415daf59c9d04dfd54b6" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40061 } } }
I20260926 12:37:54.050235 32642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:54.050580 32649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:54.050845 32641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8be234f4d94c415daf59c9d04dfd54b6. Latest consensus state: current_term: 1 leader_uuid: "8be234f4d94c415daf59c9d04dfd54b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8be234f4d94c415daf59c9d04dfd54b6" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 40061 } } }
I20260926 12:37:54.050943 32641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:54.053764 32649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:54.056854 32633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:54.048367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56985 (local address 127.10.98.62:40061)
0926 12:37:54.048472 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:54.048476 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:54.048490 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:37:54.048526 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:54.048529 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:54.048585 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:54.048658 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:54.049524 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:54.050317 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:54.051417 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:54.051725 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:54.051835 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:54.053366 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:54.053371 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:54.053374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:54.053399 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:54.055130 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:54.055583 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:54.055587 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:54.055589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:54.055679 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:54.055966 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:54.055969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:54.055971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:54.056201 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:54.056333 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:54.056571 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 12:37:54.056695 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":36}
I20260926 12:37:54.059469 32649 catalog_manager.cc:1347] Generated new cluster ID: 9ab37784108b4717a6205e69d995e369
I20260926 12:37:54.059525 32649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:54.067445 32649 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:54.068341 32649 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:54.079954 32649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8be234f4d94c415daf59c9d04dfd54b6: Generated new TSK 0
I20260926 12:37:54.080783 32649 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:37:54.090198 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 32562
2026-09-26T12:37:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:54 dist-test-slave-t7tk krb5kdc[32661](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:54 dist-test-slave-t7tk krb5kdc[32661](info): set up 2 sockets
Sep 26 12:37:54 dist-test-slave-t7tk krb5kdc[32661](info): commencing operation
krb5kdc: starting...
W20260926 12:37:55.042553 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:55 dist-test-slave-t7tk krb5kdc[32661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426275, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:37:55.160542 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:39751
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:37649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:39751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:55.270666 32674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:55.270937 32674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:55.270978 32674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:55.274969 32674 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:55.275027 32674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:55.275045 32674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:55.275061 32674 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:55.275079 32674 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:55.280354 32674 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:37649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:39751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:39751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:55.281419 32674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:55.282425 32674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:55.290966 32679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:55.291052 32680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:55.291031 32674 server_base.cc:1034] running on GCE node
W20260926 12:37:55.290963 32682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:55.291774 32674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:55.292896 32674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:55.294065 32674 hybrid_clock.cc:648] HybridClock initialized: now 1790426275294039 us; error 40 us; skew 500 ppm
Sep 26 12:37:55 dist-test-slave-t7tk krb5kdc[32661](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426275, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:55.296772 32674 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:37:55.297726 32674 webserver.cc:460] Webserver started at http://127.10.98.62:35585/ using document root <none> and password file <none>
I20260926 12:37:55.298270 32674 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:55.298329 32674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:55.298508 32674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:55.300469 32674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "f7df6b0554224a26a29b106b91dc5d11"
format_stamp: "Formatted at 2026-09-26 12:37:55 on dist-test-slave-t7tk"
server_key: "d6a6b558953aaeca704fd1394b014857"
server_key_iv: "d55ce5e0a8909c28fdfc4e5dd3f828dc"
server_key_version: "encryptionkey@0"
I20260926 12:37:55.300978 32674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "f7df6b0554224a26a29b106b91dc5d11"
format_stamp: "Formatted at 2026-09-26 12:37:55 on dist-test-slave-t7tk"
server_key: "d6a6b558953aaeca704fd1394b014857"
server_key_iv: "d55ce5e0a8909c28fdfc4e5dd3f828dc"
server_key_version: "encryptionkey@0"
I20260926 12:37:55.304553 32674 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 12:37:55.306972 32689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:55.308173 32674 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260926 12:37:55.308283 32674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "f7df6b0554224a26a29b106b91dc5d11"
format_stamp: "Formatted at 2026-09-26 12:37:55 on dist-test-slave-t7tk"
server_key: "d6a6b558953aaeca704fd1394b014857"
server_key_iv: "d55ce5e0a8909c28fdfc4e5dd3f828dc"
server_key_version: "encryptionkey@0"
I20260926 12:37:55.308374 32674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:55.332970 32674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:55.335842 32674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:55.336021 32674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:55.342796 32674 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:39751
I20260926 12:37:55.342820 32741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:39751 every 8 connection(s)
I20260926 12:37:55.343724 32674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:55.346535 32742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:55.346774 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 32674
I20260926 12:37:55.346894 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:55.347177 10632 external_mini_cluster.cc:1428] Setting key fc8c9f72bf1084e05a65fb13612b627d
I20260926 12:37:55.353592 32742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11: Bootstrap starting.
I20260926 12:37:55.356185 32742 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:55.357044 32742 log.cc:824] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:55.357125 32745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:55.348569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52473 (local address 127.10.98.62:39751)
0926 12:37:55.349004 (+ 435us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:55.349025 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:55.349070 (+ 45us) server_negotiation.cc:407] Connection header received
0926 12:37:55.349796 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:55.349819 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:55.350256 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:55.350635 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:55.351032 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:55.352498 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:55.353455 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:55.354115 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:55.354261 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:55.355114 (+ 853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:55.355120 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:55.355136 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:55.355618 (+ 482us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:55.355685 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:55.355715 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:55.355808 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:55.355952 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 12:37:55.356163 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":109,"threads_started":1}
I20260926 12:37:55.359347 32742 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11: No bootstrap required, opened a new log
I20260926 12:37:55.360144 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:37:55.362982 32742 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7df6b0554224a26a29b106b91dc5d11" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 39751 } }
I20260926 12:37:55.363221 32742 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:55.363258 32742 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7df6b0554224a26a29b106b91dc5d11, State: Initialized, Role: FOLLOWER
I20260926 12:37:55.363677 32742 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7df6b0554224a26a29b106b91dc5d11" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 39751 } }
I20260926 12:37:55.363797 32742 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:55.363853 32742 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:55.363952 32742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:55.364820 32742 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7df6b0554224a26a29b106b91dc5d11" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 39751 } }
I20260926 12:37:55.365182 32742 leader_election.cc:302] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7df6b0554224a26a29b106b91dc5d11; no voters:
I20260926 12:37:55.365473 32742 leader_election.cc:288] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:55.365554 32752 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:55.365715 32752 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11 [term 1 LEADER]: Becoming Leader. State: Replica: f7df6b0554224a26a29b106b91dc5d11, State: Running, Role: LEADER
I20260926 12:37:55.365981 32745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:55.362079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47989 (local address 127.10.98.62:39751)
0926 12:37:55.362205 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:55.362209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:55.362395 (+ 186us) server_negotiation.cc:407] Connection header received
0926 12:37:55.362501 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:55.362505 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:55.362549 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:55.362614 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:55.362875 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:55.363457 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:55.364361 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:55.364614 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:55.364705 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:55.365521 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:55.365525 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:55.365528 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:55.365541 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:55.365572 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:55.365585 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:55.365649 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:55.365793 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 12:37:55.365850 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:37:55.366206 32752 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7df6b0554224a26a29b106b91dc5d11" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 39751 } }
I20260926 12:37:55.366791 32742 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:55.368023 32754 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7df6b0554224a26a29b106b91dc5d11. Latest consensus state: current_term: 1 leader_uuid: "f7df6b0554224a26a29b106b91dc5d11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7df6b0554224a26a29b106b91dc5d11" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 39751 } } }
I20260926 12:37:55.368045 32753 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7df6b0554224a26a29b106b91dc5d11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7df6b0554224a26a29b106b91dc5d11" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 39751 } } }
I20260926 12:37:55.368140 32753 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:55.368140 32754 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:55.368530 32762 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:55.371330 32762 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:55.376771 32762 catalog_manager.cc:1347] Generated new cluster ID: 428aa3ab2cdd424ab27cf93f7b75a6bb
I20260926 12:37:55.376847 32762 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:55.402127 32762 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:55.403465 32762 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:55.415498 32762 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7df6b0554224a26a29b106b91dc5d11: Generated new TSK 0
I20260926 12:37:55.416409 32762 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:37:55.424490 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 32674
2026-09-26T12:37:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:55 dist-test-slave-t7tk krb5kdc[305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:55 dist-test-slave-t7tk krb5kdc[305](info): set up 2 sockets
Sep 26 12:37:55 dist-test-slave-t7tk krb5kdc[305](info): commencing operation
krb5kdc: starting...
W20260926 12:37:56.390774 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.947s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:56 dist-test-slave-t7tk krb5kdc[305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426276, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:37:56.516860 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:45863
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:46025
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:45863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:56.627399 318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:56.627696 318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:56.627751 318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:56.631822 318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:56.631899 318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:56.631925 318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:56.631946 318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:56.631968 318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:56.637298 318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:46025
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:45863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:45863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:56.638388 318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:56.639422 318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:56.647969 323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:56.647972 324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:56.647972 326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:56.648149 318 server_base.cc:1034] running on GCE node
I20260926 12:37:56.648823 318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:56.649974 318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:56.651145 318 hybrid_clock.cc:648] HybridClock initialized: now 1790426276651128 us; error 31 us; skew 500 ppm
Sep 26 12:37:56 dist-test-slave-t7tk krb5kdc[305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426276, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:56.653846 318 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:37:56.654759 318 webserver.cc:460] Webserver started at http://127.10.98.62:38701/ using document root <none> and password file <none>
I20260926 12:37:56.655301 318 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:56.655364 318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:56.655567 318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:56.657675 318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "b4125deff1384c1aa15b77059246221d"
format_stamp: "Formatted at 2026-09-26 12:37:56 on dist-test-slave-t7tk"
server_key: "d04ebb3b43b15a7547a1c185ec13ff59"
server_key_iv: "6f9a199a32261b2565c2fa301c9e2dc7"
server_key_version: "encryptionkey@0"
I20260926 12:37:56.658217 318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "b4125deff1384c1aa15b77059246221d"
format_stamp: "Formatted at 2026-09-26 12:37:56 on dist-test-slave-t7tk"
server_key: "d04ebb3b43b15a7547a1c185ec13ff59"
server_key_iv: "6f9a199a32261b2565c2fa301c9e2dc7"
server_key_version: "encryptionkey@0"
I20260926 12:37:56.662047 318 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:37:56.664722 333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:56.666108 318 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:37:56.666229 318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "b4125deff1384c1aa15b77059246221d"
format_stamp: "Formatted at 2026-09-26 12:37:56 on dist-test-slave-t7tk"
server_key: "d04ebb3b43b15a7547a1c185ec13ff59"
server_key_iv: "6f9a199a32261b2565c2fa301c9e2dc7"
server_key_version: "encryptionkey@0"
I20260926 12:37:56.666338 318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:56.671025 318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:56.674233 318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:56.674393 318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:56.681278 318 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:45863
I20260926 12:37:56.681300 385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:45863 every 8 connection(s)
I20260926 12:37:56.682214 318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:56.683264 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 318
I20260926 12:37:56.683408 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:56.683707 10632 external_mini_cluster.cc:1428] Setting key fa649111699b705f6d8bebafc639d573
I20260926 12:37:56.685638 386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:56.688808 389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:56.685090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52833 (local address 127.10.98.62:45863)
0926 12:37:56.685573 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:56.685590 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:56.685628 (+ 38us) server_negotiation.cc:407] Connection header received
0926 12:37:56.686289 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:56.686307 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:56.686594 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:56.686913 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:56.687668 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:56.687673 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:56.687684 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:56.687694 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:56.687745 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:56.687764 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:56.687888 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:56.687966 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 12:37:56.688121 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":148,"threads_started":1}
I20260926 12:37:56.691718 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:37:56.691745 386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d: Bootstrap starting.
I20260926 12:37:56.694036 386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:56.694912 386 log.cc:824] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:56.695364 389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:56.693718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44355 (local address 127.10.98.62:45863)
0926 12:37:56.693838 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:56.693842 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:56.693996 (+ 154us) server_negotiation.cc:407] Connection header received
0926 12:37:56.694097 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:56.694100 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:56.694137 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:56.694222 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:56.694885 (+ 663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:56.694888 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:56.694891 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 12:37:56.694902 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:56.694931 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 12:37:56.694945 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:56.695034 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:56.695207 (+ 173us) server_negotiation.cc:299] Negotiation successful
0926 12:37:56.695247 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:37:56.697062 386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d: No bootstrap required, opened a new log
I20260926 12:37:56.700536 386 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4125deff1384c1aa15b77059246221d" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45863 } }
I20260926 12:37:56.700783 386 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 12:37:56.700822 386 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4125deff1384c1aa15b77059246221d, State: Initialized, Role: FOLLOWER
I20260926 12:37:56.701238 386 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4125deff1384c1aa15b77059246221d" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45863 } }
I20260926 12:37:56.701366 386 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 12:37:56.701413 386 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 12:37:56.701485 386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:56.702479 386 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4125deff1384c1aa15b77059246221d" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45863 } }
I20260926 12:37:56.702813 386 leader_election.cc:302] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4125deff1384c1aa15b77059246221d; no voters:
I20260926 12:37:56.703112 386 leader_election.cc:288] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 12:37:56.703213 396 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:56.703419 396 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d [term 1 LEADER]: Becoming Leader. State: Replica: b4125deff1384c1aa15b77059246221d, State: Running, Role: LEADER
I20260926 12:37:56.703768 396 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4125deff1384c1aa15b77059246221d" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45863 } }
I20260926 12:37:56.704274 386 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:56.705377 397 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4125deff1384c1aa15b77059246221d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4125deff1384c1aa15b77059246221d" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45863 } } }
I20260926 12:37:56.705505 397 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:56.705809 398 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4125deff1384c1aa15b77059246221d. Latest consensus state: current_term: 1 leader_uuid: "b4125deff1384c1aa15b77059246221d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4125deff1384c1aa15b77059246221d" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 45863 } } }
I20260926 12:37:56.705884 405 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:56.705971 398 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:56.709232 405 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:56.715147 405 catalog_manager.cc:1347] Generated new cluster ID: 39ea4fc6e6814224ba3c7b019811f293
I20260926 12:37:56.715233 405 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:56.734759 405 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:56.735090 405 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:56.739157 405 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4125deff1384c1aa15b77059246221d: Generated new TSK 0
I20260926 12:37:56.739821 405 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 12:37:56.752185 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 318
2026-09-26T12:37:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1327 ms)
[----------] 18 tests from AuthTokenIssuingTest (23925 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:56 dist-test-slave-t7tk krb5kdc[417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:56 dist-test-slave-t7tk krb5kdc[417](info): set up 2 sockets
Sep 26 12:37:56 dist-test-slave-t7tk krb5kdc[417](info): commencing operation
krb5kdc: starting...
W20260926 12:37:57.722635 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.952s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:37:57 dist-test-slave-t7tk krb5kdc[417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:37:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:37:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:37:57.840477 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:35357
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:35357,127.10.98.61:35385,127.10.98.60:45767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:37:57.951550 430 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 12:37:57.951875 430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:57.951911 430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:57.951977 430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:37:57.956032 430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:57.956101 430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:57.956126 430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:57.956147 430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:57.956168 430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:57.961601 430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:35357,127.10.98.61:35385,127.10.98.60:45767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:35357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:57.962734 430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:57.963796 430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:57.972366 435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:57.972366 438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:57.972481 430 server_base.cc:1034] running on GCE node
W20260926 12:37:57.972695 436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:57.973276 430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:57.974382 430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:57.975539 430 hybrid_clock.cc:648] HybridClock initialized: now 1790426277975523 us; error 33 us; skew 500 ppm
Sep 26 12:37:57 dist-test-slave-t7tk krb5kdc[417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426277, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:57.978354 430 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:37:57.979239 430 webserver.cc:460] Webserver started at http://127.10.98.62:43781/ using document root <none> and password file <none>
I20260926 12:37:57.979779 430 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:57.979842 430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:57.980020 430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:57.982237 430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "a182024d39094415b6d46bdc1b6a7e70"
format_stamp: "Formatted at 2026-09-26 12:37:57 on dist-test-slave-t7tk"
server_key: "e3e1e47b246fb6732a386da4f6b78717"
server_key_iv: "a82b5d67e0bdec56cddaad3674f5384d"
server_key_version: "encryptionkey@0"
I20260926 12:37:57.982779 430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "a182024d39094415b6d46bdc1b6a7e70"
format_stamp: "Formatted at 2026-09-26 12:37:57 on dist-test-slave-t7tk"
server_key: "e3e1e47b246fb6732a386da4f6b78717"
server_key_iv: "a82b5d67e0bdec56cddaad3674f5384d"
server_key_version: "encryptionkey@0"
I20260926 12:37:57.986514 430 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:37:57.988935 445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:57.990244 430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260926 12:37:57.990350 430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "a182024d39094415b6d46bdc1b6a7e70"
format_stamp: "Formatted at 2026-09-26 12:37:57 on dist-test-slave-t7tk"
server_key: "e3e1e47b246fb6732a386da4f6b78717"
server_key_iv: "a82b5d67e0bdec56cddaad3674f5384d"
server_key_version: "encryptionkey@0"
I20260926 12:37:57.990439 430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:58.008294 430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:58.011417 430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:58.011584 430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:58.018610 430 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:35357
I20260926 12:37:58.018872 500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:35357 every 8 connection(s)
I20260926 12:37:58.019575 430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:37:58.022449 501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:58.026367 501 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35357 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:37:58.026885 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 430
I20260926 12:37:58.027005 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:37:58.027279 10632 external_mini_cluster.cc:1428] Setting key c9cbce510e459c590012478edc9dad3d
WARNING: no policy specified for kudu/127.10.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.61@KRBTEST.COM" created.
Sep 26 12:37:58 dist-test-slave-t7tk krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426277, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:37:58.045076 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.028917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:35357 (local address 127.0.0.1:58914)
0926 12:37:58.029428 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.029484 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:58.030406 (+ 922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:58.031240 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:58.031245 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:58.031633 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:58.031904 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:58.031919 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.033668 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.033670 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:58.034420 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:58.034429 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.035325 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.035328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:58.035443 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.036080 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:37:58.036106 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:58.039433 (+ 3327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:58.041819 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:58.041828 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:58.041838 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:58.042086 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:58.042495 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:58.042498 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:58.042547 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:58.042678 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:58.043544 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:58.043549 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:58.043688 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 12:37:58.043844 (+ 156us) client_negotiation.cc:240] Negotiation successful
0926 12:37:58.044145 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":7,"spinlock_wait_cycles":4736,"thread_start_us":158,"threads_started":1}
I20260926 12:37:58.045077 505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.029847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58914 (local address 127.10.98.62:35357)
0926 12:37:58.030140 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.030151 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.030197 (+ 46us) server_negotiation.cc:407] Connection header received
0926 12:37:58.030762 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.030768 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.031021 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.031234 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.032062 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.033521 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.034545 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.035167 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.035343 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.039619 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:58.039626 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:58.039641 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:58.039668 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:58.041651 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.042203 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.042210 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.042212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.042366 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.042780 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.042783 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.042785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.043393 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:58.043548 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:58.043971 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 12:37:58.044282 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":190,"spinlock_wait_cycles":4352,"thread_start_us":89,"threads_started":1}
I20260926 12:37:58.047775 501 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:37:58.048601 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.048246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:35385 (local address 127.0.0.1:37030)
0926 12:37:58.048393 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.048469 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.61:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260926 12:37:58.048883 451 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.98.61:35385: connect: Connection refused (error 111)
I20260926 12:37:58.049845 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.049627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:35385 (local address 127.0.0.1:37046)
0926 12:37:58.049743 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.049790 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.61:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 12:37:58.050000 501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.61:35385: Network error: Client connection negotiation failed: client connection to 127.10.98.61:35385: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.61.keytab.
Entry for principal kudu/127.10.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.61.keytab.
I20260926 12:37:58.056330 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.61:35385
--webserver_interface=127.10.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.61.keytab
--principal=kudu/127.10.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:35357,127.10.98.61:35385,127.10.98.60:45767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:37:58.099272 501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } attempt: 1
I20260926 12:37:58.100196 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.099861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:35385 (local address 127.0.0.1:37062)
0926 12:37:58.100003 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.100098 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.61:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260926 12:37:58.100922 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.100715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:35385 (local address 127.0.0.1:37068)
0926 12:37:58.100819 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.100867 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.61:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 12:37:58.101065 501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.61:35385: Network error: Client connection negotiation failed: client connection to 127.10.98.61:35385: connect: Connection refused (error 111)
W20260926 12:37:58.168561 508 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 12:37:58.168856 508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:58.168880 508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:58.168918 508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 12:37:58.169281 501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } attempt: 2
I20260926 12:37:58.170058 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.169780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:35385 (local address 127.0.0.1:37074)
0926 12:37:58.169929 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.169986 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.61:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 12:37:58.170735 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.170535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:35385 (local address 127.0.0.1:37078)
0926 12:37:58.170633 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.170679 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.61:35385: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 12:37:58.170858 501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.61:35385: Network error: Client connection negotiation failed: client connection to 127.10.98.61:35385: connect: Connection refused (error 111)
W20260926 12:37:58.173337 508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:58.173391 508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:58.173409 508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:58.173427 508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:58.173456 508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:58.178920 508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:35357,127.10.98.61:35385,127.10.98.60:45767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.61.keytab
--principal=kudu/127.10.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.61:35385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:58.180075 508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:58.181201 508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:58.190732 513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:58.190891 516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:58.190738 514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:58.191217 508 server_base.cc:1034] running on GCE node
I20260926 12:37:58.191980 508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:58.193195 508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:58.194370 508 hybrid_clock.cc:648] HybridClock initialized: now 1790426278194348 us; error 36 us; skew 500 ppm
Sep 26 12:37:58 dist-test-slave-t7tk krb5kdc[417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426278, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:58.197269 508 init.cc:377] Logged in from keytab as kudu/127.10.98.61@KRBTEST.COM (short username kudu)
I20260926 12:37:58.198242 508 webserver.cc:460] Webserver started at http://127.10.98.61:34141/ using document root <none> and password file <none>
I20260926 12:37:58.198802 508 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:58.198871 508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:58.199075 508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:58.201189 508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-1/data/instance:
uuid: "2deb20352a3549709947af6355ccd4c5"
format_stamp: "Formatted at 2026-09-26 12:37:58 on dist-test-slave-t7tk"
server_key: "332d8a67b73f18e249f268157599180d"
server_key_iv: "32ebea7b3bf570b50ca81c154c76ac49"
server_key_version: "encryptionkey@0"
I20260926 12:37:58.201746 508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-1/wal/instance:
uuid: "2deb20352a3549709947af6355ccd4c5"
format_stamp: "Formatted at 2026-09-26 12:37:58 on dist-test-slave-t7tk"
server_key: "332d8a67b73f18e249f268157599180d"
server_key_iv: "32ebea7b3bf570b50ca81c154c76ac49"
server_key_version: "encryptionkey@0"
I20260926 12:37:58.205665 508 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260926 12:37:58.208250 523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:58.209570 508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:37:58.209695 508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-1/wal
uuid: "2deb20352a3549709947af6355ccd4c5"
format_stamp: "Formatted at 2026-09-26 12:37:58 on dist-test-slave-t7tk"
server_key: "332d8a67b73f18e249f268157599180d"
server_key_iv: "32ebea7b3bf570b50ca81c154c76ac49"
server_key_version: "encryptionkey@0"
I20260926 12:37:58.209800 508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:58.229907 508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:58.233340 508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:58.233523 508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:58.240491 508 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.61:35385
I20260926 12:37:58.240746 576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.61:35385 every 8 connection(s)
I20260926 12:37:58.241524 508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-1/data/info.pb
I20260926 12:37:58.242664 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 508
I20260926 12:37:58.242779 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-1/wal/instance
I20260926 12:37:58.243037 10632 external_mini_cluster.cc:1428] Setting key 1907a04d9d1532c863d8423f5fb33227
I20260926 12:37:58.244407 577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:58.248152 577 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35357 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.60@KRBTEST.COM" created.
Sep 26 12:37:58 dist-test-slave-t7tk krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426278, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.61@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:37:58.262023 501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } attempt: 3
I20260926 12:37:58.263547 505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.249992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58922 (local address 127.10.98.62:35357)
0926 12:37:58.250141 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.250145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.250956 (+ 811us) server_negotiation.cc:407] Connection header received
0926 12:37:58.251939 (+ 983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.251942 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.252000 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.252075 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.253701 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.254242 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.255845 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.256079 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.256161 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.258929 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:58.258932 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:58.258934 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:58.258959 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:58.260312 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.260815 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.260818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.260819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.260895 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.261317 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.261320 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.261322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.261508 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:58.261602 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:58.262244 (+ 642us) server_negotiation.cc:299] Negotiation successful
0926 12:37:58.263410 (+ 1166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 12:37:58.263664 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.250331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:35357 (local address 127.0.0.1:58922)
0926 12:37:58.250811 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.250856 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:58.251737 (+ 881us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:58.252259 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:58.252266 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:58.252780 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:58.253515 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:58.253537 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.254374 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.254377 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:58.255727 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:58.255739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.256239 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.256244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:58.256450 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.257116 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:37:58.257141 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:58.258759 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:58.260422 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:58.260428 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:58.260444 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:58.260715 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:58.260976 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:58.260979 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:58.260982 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:58.261226 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:58.261609 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:58.261616 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:58.261955 (+ 339us) client_negotiation.cc:769] Sending connection context
0926 12:37:58.262129 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 12:37:58.262541 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":127,"threads_started":1}
I20260926 12:37:58.265236 577 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:37:58 dist-test-slave-t7tk krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426277, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for kudu/127.10.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.60.keytab.
Entry for principal kudu/127.10.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.60.keytab.
I20260926 12:37:58.270251 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.60:45767
--webserver_interface=127.10.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.60.keytab
--principal=kudu/127.10.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:34899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:35357,127.10.98.61:35385,127.10.98.60:45767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 12:37:58 dist-test-slave-t7tk krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426278, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.61@KRBTEST.COM for kudu/127.10.98.61@KRBTEST.COM
I20260926 12:37:58.273523 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.262464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:35385 (local address 127.0.0.1:37084)
0926 12:37:58.263025 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.263040 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:58.263196 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:58.263958 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:58.263961 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:58.264133 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:58.264249 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:58.264257 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.265448 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.265452 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:58.266653 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:58.266664 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.267251 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.267255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:58.267358 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.267888 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:37:58.267900 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:58.269218 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:58.271577 (+ 2359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:58.271581 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:58.271583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:58.271845 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:58.272269 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:58.272272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:58.272274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:58.272374 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:58.272954 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:58.272957 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:58.273077 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 12:37:58.273218 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 12:37:58.273358 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482}
I20260926 12:37:58.273703 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.262674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37084 (local address 127.10.98.61:35385)
0926 12:37:58.263066 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.263076 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.263113 (+ 37us) server_negotiation.cc:407] Connection header received
0926 12:37:58.263296 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.263301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.263469 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.263600 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.264356 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.265337 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.266746 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.267139 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.267238 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.269352 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:58.269360 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:58.269373 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:58.269394 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:58.271429 (+ 2035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.271997 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.272003 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.272004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.272146 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.272490 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.272493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.272494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.272831 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:58.272987 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:58.273307 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 12:37:58.273503 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":137,"threads_started":1}
I20260926 12:37:58.274786 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.265693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:35385 (local address 127.0.0.1:37098)
0926 12:37:58.265860 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.265875 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:58.265945 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:58.266310 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:58.266313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:58.266482 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:58.266585 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:58.266592 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.267375 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.267378 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:58.268092 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:58.268101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.268508 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.268512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:58.268626 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.269440 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:37:58.269456 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:58.270850 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:58.273377 (+ 2527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:58.273379 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:58.273381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:58.273633 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:58.273894 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:58.273897 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:58.273898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:58.273971 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:58.274355 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:58.274357 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:58.274426 (+ 69us) client_negotiation.cc:769] Sending connection context
0926 12:37:58.274510 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 12:37:58.274643 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":15}
I20260926 12:37:58.275218 583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.265774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37098 (local address 127.10.98.61:35385)
0926 12:37:58.266060 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.266067 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.266098 (+ 31us) server_negotiation.cc:407] Connection header received
0926 12:37:58.266132 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.266135 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.266200 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.266269 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.266711 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.267284 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.268195 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.268420 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.268498 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.270979 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:58.270983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:58.270985 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:58.271007 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:58.273281 (+ 2274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.273728 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.273731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.273733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.273803 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.274047 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.274049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.274051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.274239 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:58.274317 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:58.274962 (+ 645us) server_negotiation.cc:299] Negotiation successful
0926 12:37:58.275066 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":111,"threads_started":1}
I20260926 12:37:58.275913 501 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:37:58.276326 577 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:37:58.276842 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.276420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:45767 (local address 127.0.0.1:41562)
0926 12:37:58.276696 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.276763 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:45767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260926 12:37:58.276973 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.276712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:45767 (local address 127.0.0.1:41578)
0926 12:37:58.276819 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.276889 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:45767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 12:37:58.277375 524 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.98.60:45767: connect: Connection refused (error 111)
I20260926 12:37:58.277709 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.277406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:45767 (local address 127.0.0.1:41580)
0926 12:37:58.277587 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.277642 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:45767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260926 12:37:58.277837 501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.60:45767: Network error: Client connection negotiation failed: client connection to 127.10.98.60:45767: connect: Connection refused (error 111)
I20260926 12:37:58.278193 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.277990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:45767 (local address 127.0.0.1:41582)
0926 12:37:58.278089 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.278138 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:45767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 12:37:58.278339 577 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.60:45767: Network error: Client connection negotiation failed: client connection to 127.10.98.60:45767: connect: Connection refused (error 111)
I20260926 12:37:58.309011 501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } attempt: 1
I20260926 12:37:58.310031 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.309678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:45767 (local address 127.0.0.1:41586)
0926 12:37:58.309848 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.309934 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:45767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 12:37:58.310801 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.310576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:45767 (local address 127.0.0.1:41590)
0926 12:37:58.310686 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.310739 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:45767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 12:37:58.310943 501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.60:45767: Network error: Client connection negotiation failed: client connection to 127.10.98.60:45767: connect: Connection refused (error 111)
I20260926 12:37:58.327646 577 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } attempt: 1
I20260926 12:37:58.328567 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.328242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:45767 (local address 127.0.0.1:41598)
0926 12:37:58.328389 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.328479 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:45767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 12:37:58.329380 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.329177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:45767 (local address 127.0.0.1:41602)
0926 12:37:58.329273 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.329320 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:45767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 12:37:58.329541 577 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.60:45767: Network error: Client connection negotiation failed: client connection to 127.10.98.60:45767: connect: Connection refused (error 111)
I20260926 12:37:58.386096 501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } attempt: 2
I20260926 12:37:58.387018 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.386696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:45767 (local address 127.0.0.1:41608)
0926 12:37:58.386857 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.386928 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:45767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260926 12:37:58.386955 584 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 12:37:58.387331 584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:37:58.387362 584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:37:58.387401 584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 12:37:58.387835 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.387615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:45767 (local address 127.0.0.1:41620)
0926 12:37:58.387723 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.387775 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:45767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 12:37:58.387981 501 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.60:45767: Network error: Client connection negotiation failed: client connection to 127.10.98.60:45767: connect: Connection refused (error 111)
W20260926 12:37:58.391782 584 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:37:58.391853 584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:37:58.391880 584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:37:58.391896 584 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:37:58.391911 584 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:37:58.397622 584 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:34899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:35357,127.10.98.61:35385,127.10.98.60:45767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.60.keytab
--principal=kudu/127.10.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.60:45767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.584
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:37:58.397715 577 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } attempt: 2
I20260926 12:37:58.398697 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.398267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:45767 (local address 127.0.0.1:41630)
0926 12:37:58.398538 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.398609 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:45767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260926 12:37:58.398972 584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:37:58.399524 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.399270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:45767 (local address 127.0.0.1:41640)
0926 12:37:58.399402 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.399456 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:45767: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260926 12:37:58.399650 577 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.60:45767: Network error: Client connection negotiation failed: client connection to 127.10.98.60:45767: connect: Connection refused (error 111)
I20260926 12:37:58.400053 584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 12:37:58.408641 593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:58.408641 590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:37:58.408775 591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:37:58.408874 584 server_base.cc:1034] running on GCE node
I20260926 12:37:58.409744 584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:37:58.410995 584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:37:58.412165 584 hybrid_clock.cc:648] HybridClock initialized: now 1790426278412138 us; error 48 us; skew 500 ppm
Sep 26 12:37:58 dist-test-slave-t7tk krb5kdc[417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426278, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:37:58.415197 584 init.cc:377] Logged in from keytab as kudu/127.10.98.60@KRBTEST.COM (short username kudu)
I20260926 12:37:58.416123 584 webserver.cc:460] Webserver started at http://127.10.98.60:41733/ using document root <none> and password file <none>
I20260926 12:37:58.416694 584 fs_manager.cc:359] Metadata directory not provided
I20260926 12:37:58.416762 584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:37:58.416954 584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:37:58.419147 584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-2/data/instance:
uuid: "c3ec323bdcdb480280dcc17ce967d680"
format_stamp: "Formatted at 2026-09-26 12:37:58 on dist-test-slave-t7tk"
server_key: "7878dc60ad280f62830115193266f05c"
server_key_iv: "2a98fc936aed426340fc534dfcab7c34"
server_key_version: "encryptionkey@0"
I20260926 12:37:58.419747 584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-2/wal/instance:
uuid: "c3ec323bdcdb480280dcc17ce967d680"
format_stamp: "Formatted at 2026-09-26 12:37:58 on dist-test-slave-t7tk"
server_key: "7878dc60ad280f62830115193266f05c"
server_key_iv: "2a98fc936aed426340fc534dfcab7c34"
server_key_version: "encryptionkey@0"
I20260926 12:37:58.423656 584 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260926 12:37:58.426246 600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:37:58.427565 584 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:37:58.427712 584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-2/wal
uuid: "c3ec323bdcdb480280dcc17ce967d680"
format_stamp: "Formatted at 2026-09-26 12:37:58 on dist-test-slave-t7tk"
server_key: "7878dc60ad280f62830115193266f05c"
server_key_iv: "2a98fc936aed426340fc534dfcab7c34"
server_key_version: "encryptionkey@0"
I20260926 12:37:58.427820 584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:37:58.443823 584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:37:58.446966 584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:37:58.447163 584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:37:58.453853 584 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.60:45767
I20260926 12:37:58.454175 652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.60:45767 every 8 connection(s)
I20260926 12:37:58.454931 584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-2/data/info.pb
I20260926 12:37:58.457836 653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:37:58.457998 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 584
I20260926 12:37:58.458120 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790425997379116-10632-0/minicluster-data/master-2/wal/instance
I20260926 12:37:58.458424 10632 external_mini_cluster.cc:1428] Setting key 5252f64a87022548a92b3f33184cda76
I20260926 12:37:58.463011 653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35357 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:37:58 dist-test-slave-t7tk krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.60@KRBTEST.COM
Sep 26 12:37:58 dist-test-slave-t7tk krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426278, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.60@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:37:58.474069 656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.459923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41646 (local address 127.10.98.60:45767)
0926 12:37:58.460390 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.460410 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.460552 (+ 142us) server_negotiation.cc:407] Connection header received
0926 12:37:58.461369 (+ 817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.461396 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.461869 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.462257 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.462973 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.464442 (+ 1469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.465514 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.466163 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.466285 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.468885 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:58.468891 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:58.468903 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:58.468931 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:58.470747 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.471195 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.471201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.471208 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.471341 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.471647 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.471651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.471653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.472204 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:58.472424 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:58.472656 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 12:37:58.472998 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":303,"thread_start_us":128,"threads_started":1}
I20260926 12:37:58.474543 657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.465224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:35357 (local address 127.0.0.1:58928)
0926 12:37:58.465629 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.465672 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:58.465789 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:58.466049 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:58.466056 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:58.466519 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:58.466683 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:58.466690 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.467383 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.467386 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:58.468224 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:58.468236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.468814 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.468819 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:58.468923 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.469409 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:37:58.469420 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:58.470901 (+ 1481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:58.472831 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:58.472838 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:58.472844 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:58.473172 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:58.473434 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:58.473437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:58.473438 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:58.473523 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:58.473912 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:58.473918 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:58.474044 (+ 126us) client_negotiation.cc:769] Sending connection context
0926 12:37:58.474185 (+ 141us) client_negotiation.cc:240] Negotiation successful
0926 12:37:58.474364 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":130,"threads_started":1}
I20260926 12:37:58.474566 505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.465254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58928 (local address 127.10.98.62:35357)
0926 12:37:58.465405 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.465409 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.465748 (+ 339us) server_negotiation.cc:407] Connection header received
0926 12:37:58.465874 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.465877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.465922 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.465986 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.466794 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.467267 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.468358 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.468645 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.468737 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.471047 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:58.471050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:58.471052 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:58.471079 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:58.472712 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.473265 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.473267 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.473269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.473348 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.473630 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.473633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.473634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.473821 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:58.473893 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:58.474316 (+ 423us) server_negotiation.cc:299] Negotiation successful
0926 12:37:58.474441 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 12:37:58.476449 653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:37:58 dist-test-slave-t7tk krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426278, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.60@KRBTEST.COM for kudu/127.10.98.61@KRBTEST.COM
Sep 26 12:37:58 dist-test-slave-t7tk krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:37:58.484768 657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.476923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:35385 (local address 127.0.0.1:37100)
0926 12:37:58.477076 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.477089 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:58.477177 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:58.477407 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:58.477409 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:58.477546 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:58.477648 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:58.477653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.478415 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.478419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:58.478950 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:58.478958 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.479478 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.479482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:58.479585 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.480096 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:37:58.480108 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:58.481409 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:58.483224 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:58.483227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:58.483230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:58.483457 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:58.483770 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:58.483773 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:58.483775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:58.483864 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:58.484269 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:58.484272 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:58.484374 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 12:37:58.484461 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 12:37:58.484596 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260926 12:37:58.484839 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.477027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37100 (local address 127.10.98.61:35385)
0926 12:37:58.477170 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.477174 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.477187 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:37:58.477231 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.477234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.477280 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.477361 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.477773 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.478317 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.479071 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.479359 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.479440 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.481554 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:58.481557 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:58.481560 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:58.481593 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:58.483113 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.483547 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.483549 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.483550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.483691 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.483948 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.483951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.483952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.484187 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:58.484261 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:58.484537 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 12:37:58.484704 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 12:37:58.485612 653 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:37:58.485689 505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.476477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58934 (local address 127.10.98.62:35357)
0926 12:37:58.476821 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.476825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.476838 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:37:58.476874 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.476877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.476918 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.476989 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.477880 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.478373 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.479254 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.479510 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.479615 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.482124 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:58.482128 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:58.482130 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:58.482158 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:58.483744 (+ 1586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.484202 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.484205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.484206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.484296 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.484548 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.484552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.484553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.484840 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:58.485060 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:58.485442 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 12:37:58.485549 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259}
I20260926 12:37:58.487123 501 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } attempt: 3
Sep 26 12:37:58 dist-test-slave-t7tk krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426278, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.60@KRBTEST.COM for kudu/127.10.98.60@KRBTEST.COM
I20260926 12:37:58.490787 577 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } attempt: 3
Sep 26 12:37:58 dist-test-slave-t7tk krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426277, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.61@KRBTEST.COM
Sep 26 12:37:58 dist-test-slave-t7tk krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426277, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for kudu/127.10.98.60@KRBTEST.COM
I20260926 12:37:58.494238 657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.486111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:45767 (local address 127.0.0.1:41648)
0926 12:37:58.486348 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.486362 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:58.486445 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:58.486639 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:58.486642 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:58.486794 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:58.486886 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:58.486892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.487761 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.487765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:58.488569 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:58.488579 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.489033 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.489036 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:58.489131 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.489540 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:37:58.489553 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:58.490879 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:58.492511 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:58.492514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:58.492517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:58.492727 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:58.492959 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:58.492963 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:58.492964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:58.493036 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:58.493479 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:58.493481 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:58.493544 (+ 63us) client_negotiation.cc:769] Sending connection context
0926 12:37:58.493644 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 12:37:58.494055 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260926 12:37:58.495713 656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.486214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41648 (local address 127.10.98.60:45767)
0926 12:37:58.486314 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.486318 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.486418 (+ 100us) server_negotiation.cc:407] Connection header received
0926 12:37:58.486513 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.486516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.486558 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.486633 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.486974 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.487683 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.488689 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.488940 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.489019 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.491005 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:58.491009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:58.491011 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:58.491032 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:58.492437 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.492810 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.492813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.492815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.492885 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.493184 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.493188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.493189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.493365 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:58.493451 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:58.493784 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 12:37:58.495551 (+ 1767us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1653,"server-negotiator.queue_time_us":36}
I20260926 12:37:58.497069 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.487509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:45767 (local address 127.0.0.1:41654)
0926 12:37:58.487676 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.487690 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:58.487757 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:58.488190 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:58.488193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:58.488336 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:58.488449 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:58.488457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.489881 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.489884 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:58.490543 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:58.490552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.491125 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.491129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:58.491219 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.491599 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:37:58.491614 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:58.493462 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:58.495552 (+ 2090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:58.495556 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:58.495557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:58.495803 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:58.496093 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:58.496096 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:58.496098 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:58.496181 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:58.496601 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:58.496604 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:58.496705 (+ 101us) client_negotiation.cc:769] Sending connection context
0926 12:37:58.496798 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 12:37:58.496926 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
Sep 26 12:37:58 dist-test-slave-t7tk krb5kdc[417](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426278, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.61@KRBTEST.COM for kudu/127.10.98.60@KRBTEST.COM
I20260926 12:37:58.497781 658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.487534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41654 (local address 127.10.98.60:45767)
0926 12:37:58.487998 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.488004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.488014 (+ 10us) server_negotiation.cc:407] Connection header received
0926 12:37:58.488039 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.488041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.488070 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.488150 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.488570 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.489786 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.490664 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.490981 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.491421 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.493584 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:58.493588 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:58.493590 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:58.493609 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:58.495386 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.495910 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.495913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.495915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.496007 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.496279 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.496282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.496284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.496465 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:58.496542 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:58.497504 (+ 962us) server_negotiation.cc:299] Negotiation successful
0926 12:37:58.497619 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"spinlock_wait_cycles":6912,"thread_start_us":96,"threads_started":1}
I20260926 12:37:58.498023 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.486873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37110 (local address 127.10.98.61:35385)
0926 12:37:58.486975 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.486978 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.486990 (+ 12us) server_negotiation.cc:407] Connection header received
0926 12:37:58.487100 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.487103 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.487141 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.487202 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.487892 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.488636 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.489750 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.490074 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.490192 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.494099 (+ 3907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:58.494103 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:58.494106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:58.494133 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:58.496088 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.496571 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.496575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.496577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.496670 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.496977 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.496981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.496982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.497210 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:58.497398 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:58.497766 (+ 368us) server_negotiation.cc:299] Negotiation successful
0926 12:37:58.497872 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 12:37:58.499199 653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3ec323bdcdb480280dcc17ce967d680: Bootstrap starting.
I20260926 12:37:58.500622 501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70: Bootstrap starting.
I20260926 12:37:58.501623 653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c3ec323bdcdb480280dcc17ce967d680: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:58.502324 653 log.cc:824] T 00000000000000000000000000000000 P c3ec323bdcdb480280dcc17ce967d680: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:58.502969 501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:58.503685 501 log.cc:824] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70: Log is configured to *not* fsync() on all Append() calls
I20260926 12:37:58.504319 653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c3ec323bdcdb480280dcc17ce967d680: No bootstrap required, opened a new log
I20260926 12:37:58.506125 501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70: No bootstrap required, opened a new log
I20260926 12:37:58.507306 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:37:58.507881 580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.491183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:45767 (local address 127.0.0.1:41668)
0926 12:37:58.492283 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
0926 12:37:58.492297 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:37:58.492386 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:37:58.492646 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:37:58.492649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:37:58.492832 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:37:58.492938 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:58.492945 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.494174 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.494177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:58.494908 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:37:58.494920 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.495599 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.495604 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:37:58.495703 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.496244 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:37:58.496258 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:37:58.497707 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:37:58.506207 (+ 8500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:58.506211 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:58.506213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:58.506470 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:58.506818 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:37:58.506821 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:37:58.506823 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:37:58.506923 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:37:58.507380 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:37:58.507383 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:37:58.507485 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 12:37:58.507562 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 12:37:58.507714 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1031}
I20260926 12:37:58.508553 659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.491919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41668 (local address 127.10.98.60:45767)
0926 12:37:58.492227 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.492235 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.492364 (+ 129us) server_negotiation.cc:407] Connection header received
0926 12:37:58.492457 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.492460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.492502 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.493046 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.493205 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.494062 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.495112 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.495455 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.495899 (+ 444us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.497845 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:58.497849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:58.497851 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:58.497873 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:58.506058 (+ 8185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.506608 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.506611 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.506613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.506719 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.507037 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.507040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.507042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.507275 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:58.508164 (+ 889us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:58.508259 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 12:37:58.508385 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":237,"thread_start_us":135,"threads_started":1}
I20260926 12:37:58.508831 475 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "a182024d39094415b6d46bdc1b6a7e70"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58934
I20260926 12:37:58.509577 653 raft_consensus.cc:348] T 00000000000000000000000000000000 P c3ec323bdcdb480280dcc17ce967d680 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a182024d39094415b6d46bdc1b6a7e70" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35357 } } peers { permanent_uuid: "2deb20352a3549709947af6355ccd4c5" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } } peers { permanent_uuid: "c3ec323bdcdb480280dcc17ce967d680" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } }
I20260926 12:37:58.509836 653 raft_consensus.cc:729] T 00000000000000000000000000000000 P c3ec323bdcdb480280dcc17ce967d680 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3ec323bdcdb480280dcc17ce967d680, State: Initialized, Role: FOLLOWER
I20260926 12:37:58.510181 653 consensus_queue.cc:260] T 00000000000000000000000000000000 P c3ec323bdcdb480280dcc17ce967d680 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a182024d39094415b6d46bdc1b6a7e70" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35357 } } peers { permanent_uuid: "2deb20352a3549709947af6355ccd4c5" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } } peers { permanent_uuid: "c3ec323bdcdb480280dcc17ce967d680" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } }
I20260926 12:37:58.510555 501 raft_consensus.cc:348] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a182024d39094415b6d46bdc1b6a7e70" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35357 } } peers { permanent_uuid: "2deb20352a3549709947af6355ccd4c5" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } } peers { permanent_uuid: "c3ec323bdcdb480280dcc17ce967d680" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } }
I20260926 12:37:58.510768 501 raft_consensus.cc:729] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a182024d39094415b6d46bdc1b6a7e70, State: Initialized, Role: FOLLOWER
I20260926 12:37:58.511060 501 consensus_queue.cc:260] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a182024d39094415b6d46bdc1b6a7e70" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35357 } } peers { permanent_uuid: "2deb20352a3549709947af6355ccd4c5" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } } peers { permanent_uuid: "c3ec323bdcdb480280dcc17ce967d680" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } }
I20260926 12:37:58.511601 577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2deb20352a3549709947af6355ccd4c5: Bootstrap starting.
I20260926 12:37:58.511620 653 sys_catalog.cc:564] T 00000000000000000000000000000000 P c3ec323bdcdb480280dcc17ce967d680 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:58.511608 663 sys_catalog.cc:455] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a182024d39094415b6d46bdc1b6a7e70" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35357 } } peers { permanent_uuid: "2deb20352a3549709947af6355ccd4c5" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } } peers { permanent_uuid: "c3ec323bdcdb480280dcc17ce967d680" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } } }
I20260926 12:37:58.511942 663 sys_catalog.cc:458] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:37:58.512329 501 sys_catalog.cc:564] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:58.512423 475 raft_consensus.cc:482] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 12:37:58.512533 475 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:58.513916 475 raft_consensus.cc:504] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a182024d39094415b6d46bdc1b6a7e70" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35357 } } peers { permanent_uuid: "2deb20352a3549709947af6355ccd4c5" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } } peers { permanent_uuid: "c3ec323bdcdb480280dcc17ce967d680" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } }
I20260926 12:37:58.514623 577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2deb20352a3549709947af6355ccd4c5: Neither blocks nor log segments found. Creating new log.
I20260926 12:37:58.515537 577 log.cc:824] T 00000000000000000000000000000000 P 2deb20352a3549709947af6355ccd4c5: Log is configured to *not* fsync() on all Append() calls
W20260926 12:37:58.517573 685 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:37:58.517644 685 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:37:58.517815 662 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3ec323bdcdb480280dcc17ce967d680 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a182024d39094415b6d46bdc1b6a7e70" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35357 } } peers { permanent_uuid: "2deb20352a3549709947af6355ccd4c5" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } } peers { permanent_uuid: "c3ec323bdcdb480280dcc17ce967d680" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } } }
I20260926 12:37:58.517943 662 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3ec323bdcdb480280dcc17ce967d680 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:37:58.516227 627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a182024d39094415b6d46bdc1b6a7e70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c3ec323bdcdb480280dcc17ce967d680"
I20260926 12:37:58.519174 627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c3ec323bdcdb480280dcc17ce967d680 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:58.520519 627 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c3ec323bdcdb480280dcc17ce967d680 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a182024d39094415b6d46bdc1b6a7e70 in term 1.
I20260926 12:37:58.520941 446 leader_election.cc:302] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a182024d39094415b6d46bdc1b6a7e70, c3ec323bdcdb480280dcc17ce967d680; no voters:
I20260926 12:37:58.521209 430 master_runner.cc:186] Error getting master registration for 127.10.98.61:35385: OK, instance_id { permanent_uuid: "2deb20352a3549709947af6355ccd4c5" instance_seqno: 1790426278238442 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260926 12:37:58.515120 453 tablet.cc:2369] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 12:37:58.524500 663 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:37:58.525350 663 raft_consensus.cc:686] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [term 1 LEADER]: Becoming Leader. State: Replica: a182024d39094415b6d46bdc1b6a7e70, State: Running, Role: LEADER
I20260926 12:37:58.525713 663 consensus_queue.cc:237] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a182024d39094415b6d46bdc1b6a7e70" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35357 } } peers { permanent_uuid: "2deb20352a3549709947af6355ccd4c5" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } } peers { permanent_uuid: "c3ec323bdcdb480280dcc17ce967d680" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } }
I20260926 12:37:58.528566 584 master_runner.cc:186] Error getting master registration for 127.10.98.61:35385: OK, instance_id { permanent_uuid: "2deb20352a3549709947af6355ccd4c5" instance_seqno: 1790426278238442 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 12:37:58.531975 584 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 12:37:58.514905 475 leader_election.cc:288] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [CANDIDATE]: Term 1 election: Requested vote from peers 2deb20352a3549709947af6355ccd4c5 (127.10.98.61:35385), c3ec323bdcdb480280dcc17ce967d680 (127.10.98.60:45767)
I20260926 12:37:58.532459 430 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 12:37:58.532986 690 sys_catalog.cc:455] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a182024d39094415b6d46bdc1b6a7e70. Latest consensus state: current_term: 1 leader_uuid: "a182024d39094415b6d46bdc1b6a7e70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a182024d39094415b6d46bdc1b6a7e70" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35357 } } peers { permanent_uuid: "2deb20352a3549709947af6355ccd4c5" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } } peers { permanent_uuid: "c3ec323bdcdb480280dcc17ce967d680" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } } }
I20260926 12:37:58.533190 690 sys_catalog.cc:458] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [sys.catalog]: This master's current role is: LEADER
W20260926 12:37:58.517372 679 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c3ec323bdcdb480280dcc17ce967d680: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:37:58.533422 679 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:37:58.533818 694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:37:58.534794 694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:37:58.542757 627 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c3ec323bdcdb480280dcc17ce967d680 [term 1 FOLLOWER]: Refusing update from remote peer a182024d39094415b6d46bdc1b6a7e70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:37:58.545267 577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2deb20352a3549709947af6355ccd4c5: No bootstrap required, opened a new log
I20260926 12:37:58.549387 690 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3ec323bdcdb480280dcc17ce967d680" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:37:58.555848 662 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3ec323bdcdb480280dcc17ce967d680 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a182024d39094415b6d46bdc1b6a7e70. Latest consensus state: current_term: 1 leader_uuid: "a182024d39094415b6d46bdc1b6a7e70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a182024d39094415b6d46bdc1b6a7e70" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35357 } } peers { permanent_uuid: "2deb20352a3549709947af6355ccd4c5" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } } peers { permanent_uuid: "c3ec323bdcdb480280dcc17ce967d680" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } } }
I20260926 12:37:58.556012 662 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3ec323bdcdb480280dcc17ce967d680 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:37:58.557325 659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.538786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41676 (local address 127.10.98.60:45767)
0926 12:37:58.538900 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.538904 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.540881 (+ 1977us) server_negotiation.cc:407] Connection header received
0926 12:37:58.542153 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.542156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.542221 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.542273 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.547477 (+ 5204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.548312 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.551299 (+ 2987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.551659 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.551741 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.553697 (+ 1956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:58.553702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:58.553706 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:58.553736 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:58.555337 (+ 1601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.555878 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.555882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.555883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.555968 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.556280 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.556283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.556285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.556542 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:58.556658 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:58.557007 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 12:37:58.557172 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:37:58.557158 577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2deb20352a3549709947af6355ccd4c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a182024d39094415b6d46bdc1b6a7e70" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35357 } } peers { permanent_uuid: "2deb20352a3549709947af6355ccd4c5" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } } peers { permanent_uuid: "c3ec323bdcdb480280dcc17ce967d680" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } }
I20260926 12:37:58.557896 577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2deb20352a3549709947af6355ccd4c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2deb20352a3549709947af6355ccd4c5, State: Initialized, Role: FOLLOWER
I20260926 12:37:58.558511 577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2deb20352a3549709947af6355ccd4c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a182024d39094415b6d46bdc1b6a7e70" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35357 } } peers { permanent_uuid: "2deb20352a3549709947af6355ccd4c5" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } } peers { permanent_uuid: "c3ec323bdcdb480280dcc17ce967d680" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } }
I20260926 12:37:58.559696 703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2deb20352a3549709947af6355ccd4c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a182024d39094415b6d46bdc1b6a7e70" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35357 } } peers { permanent_uuid: "2deb20352a3549709947af6355ccd4c5" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } } peers { permanent_uuid: "c3ec323bdcdb480280dcc17ce967d680" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } } }
I20260926 12:37:58.559937 703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2deb20352a3549709947af6355ccd4c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:37:58.560389 577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2deb20352a3549709947af6355ccd4c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:37:58.560422 550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a182024d39094415b6d46bdc1b6a7e70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2deb20352a3549709947af6355ccd4c5"
I20260926 12:37:58.560655 550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2deb20352a3549709947af6355ccd4c5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:37:58.562140 550 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2deb20352a3549709947af6355ccd4c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a182024d39094415b6d46bdc1b6a7e70 in term 1.
I20260926 12:37:58.563829 550 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2deb20352a3549709947af6355ccd4c5 [term 1 FOLLOWER]: Refusing update from remote peer a182024d39094415b6d46bdc1b6a7e70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:37:58.564404 690 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2deb20352a3549709947af6355ccd4c5" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:37:58.566939 662 sys_catalog.cc:455] T 00000000000000000000000000000000 P c3ec323bdcdb480280dcc17ce967d680 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a182024d39094415b6d46bdc1b6a7e70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a182024d39094415b6d46bdc1b6a7e70" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35357 } } peers { permanent_uuid: "2deb20352a3549709947af6355ccd4c5" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } } peers { permanent_uuid: "c3ec323bdcdb480280dcc17ce967d680" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } } }
I20260926 12:37:58.567072 662 sys_catalog.cc:458] T 00000000000000000000000000000000 P c3ec323bdcdb480280dcc17ce967d680 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:37:58.567572 697 mvcc.cc:205] Tried to move back new op lower bound from 7333586036899348480 to 7333586036844851200. Current Snapshot: MvccSnapshot[applied={T|T < 7333586036899348480}]
I20260926 12:37:58.568125 690 sys_catalog.cc:455] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a182024d39094415b6d46bdc1b6a7e70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a182024d39094415b6d46bdc1b6a7e70" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35357 } } peers { permanent_uuid: "2deb20352a3549709947af6355ccd4c5" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } } peers { permanent_uuid: "c3ec323bdcdb480280dcc17ce967d680" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } } }
I20260926 12:37:58.568244 690 sys_catalog.cc:458] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:58.568518 505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.537303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58936 (local address 127.10.98.62:35357)
0926 12:37:58.537416 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.537420 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.540572 (+ 3152us) server_negotiation.cc:407] Connection header received
0926 12:37:58.540962 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.540966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.541015 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.541065 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.545604 (+ 4539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.546314 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.550435 (+ 4121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.550719 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.550792 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.564339 (+ 13547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:58.564344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:58.564347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:58.564381 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:58.566116 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.566620 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.566624 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.566626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.566720 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.567464 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.567468 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.567470 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.567707 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:58.567829 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:58.568245 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 12:37:58.568372 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 12:37:58.573076 703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2deb20352a3549709947af6355ccd4c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a182024d39094415b6d46bdc1b6a7e70. Latest consensus state: current_term: 1 leader_uuid: "a182024d39094415b6d46bdc1b6a7e70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a182024d39094415b6d46bdc1b6a7e70" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35357 } } peers { permanent_uuid: "2deb20352a3549709947af6355ccd4c5" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } } peers { permanent_uuid: "c3ec323bdcdb480280dcc17ce967d680" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } } }
I20260926 12:37:58.573218 703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2deb20352a3549709947af6355ccd4c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:37:58.574797 694 catalog_manager.cc:1347] Generated new cluster ID: 08abc4edac0f40d9b4ff5f10ecfe8f78
I20260926 12:37:58.574869 694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:37:58.575501 582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:58.537815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37118 (local address 127.10.98.61:35385)
0926 12:37:58.537930 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.537934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.539077 (+ 1143us) server_negotiation.cc:407] Connection header received
0926 12:37:58.539295 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.539299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.539360 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.539415 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.543764 (+ 4349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.544482 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.549204 (+ 4722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:58.549508 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:58.549593 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:58.571803 (+ 22210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:37:58.571807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:37:58.571810 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:37:58.571837 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:37:58.573487 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.574052 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.574056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.574058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.574143 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:37:58.574439 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:37:58.574442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:37:58.574444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:37:58.574660 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:37:58.574768 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:58.575235 (+ 467us) server_negotiation.cc:299] Negotiation successful
0926 12:37:58.575354 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260926 12:37:58.579167 714 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2deb20352a3549709947af6355ccd4c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:37:58.579977 714 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:37:58.590191 718 sys_catalog.cc:455] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a182024d39094415b6d46bdc1b6a7e70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a182024d39094415b6d46bdc1b6a7e70" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35357 } } peers { permanent_uuid: "2deb20352a3549709947af6355ccd4c5" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } } peers { permanent_uuid: "c3ec323bdcdb480280dcc17ce967d680" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } } }
I20260926 12:37:58.590356 718 sys_catalog.cc:458] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70 [sys.catalog]: This master's current role is: LEADER
I20260926 12:37:58.591590 703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2deb20352a3549709947af6355ccd4c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a182024d39094415b6d46bdc1b6a7e70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a182024d39094415b6d46bdc1b6a7e70" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 35357 } } peers { permanent_uuid: "2deb20352a3549709947af6355ccd4c5" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 35385 } } peers { permanent_uuid: "c3ec323bdcdb480280dcc17ce967d680" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 45767 } } }
I20260926 12:37:58.591722 703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2deb20352a3549709947af6355ccd4c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:37:58.592437 719 mvcc.cc:205] Tried to move back new op lower bound from 7333586036899348480 to 7333586036844851200. Current Snapshot: MvccSnapshot[applied={T|T < 7333586036899348480}]
I20260926 12:37:58.629120 694 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:37:58.631281 694 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:37:58.681010 694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a182024d39094415b6d46bdc1b6a7e70: Generated new TSK 0
I20260926 12:37:58.681974 694 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 12:37:58.711184 505 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:37:58.708972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53424 (local address 127.10.98.62:35357)
0926 12:37:58.709132 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.709137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.710060 (+ 923us) server_negotiation.cc:407] Connection header received
0926 12:37:58.710258 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.710262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.710316 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.710371 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.711043 (+ 672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53424: BlockingRecv error: recv got EOF from 127.0.0.1:53424 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 12:37:58.712632 659 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:37:58.709458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53318 (local address 127.10.98.60:45767)
0926 12:37:58.709568 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.709584 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.711687 (+ 2103us) server_negotiation.cc:407] Connection header received
0926 12:37:58.711854 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.711857 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.711923 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.711972 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.712486 (+ 514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53318: BlockingRecv error: recv got EOF from 127.0.0.1:53318 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 12:37:58.714314 582 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:37:58.713155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58886 (local address 127.10.98.61:35385)
0926 12:37:58.713258 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.713262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.713421 (+ 159us) server_negotiation.cc:407] Connection header received
0926 12:37:58.713600 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.713604 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.713653 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.713700 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.714181 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58886: BlockingRecv error: recv got EOF from 127.0.0.1:58886 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260926 12:37:58.721833 582 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:37:58.719341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58898 (local address 127.10.98.61:35385)
0926 12:37:58.719450 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.719455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.720619 (+ 1164us) server_negotiation.cc:407] Connection header received
0926 12:37:58.720870 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.720874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.720949 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.721004 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.721724 (+ 720us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58898: BlockingRecv error: recv got EOF from 127.0.0.1:58898 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 12:37:58.731807 582 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:37:58.730031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58908 (local address 127.10.98.61:35385)
0926 12:37:58.730153 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.730157 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.730746 (+ 589us) server_negotiation.cc:407] Connection header received
0926 12:37:58.731006 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.731010 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.731064 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.731117 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.731681 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58908: BlockingRecv error: recv got EOF from 127.0.0.1:58908 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 12:37:58.737442 582 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:37:58.736337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58924 (local address 127.10.98.61:35385)
0926 12:37:58.736455 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.736459 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.736648 (+ 189us) server_negotiation.cc:407] Connection header received
0926 12:37:58.736738 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.736740 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.736784 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.736851 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.737322 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58924: BlockingRecv error: recv got EOF from 127.0.0.1:58924 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 12:37:58.744733 582 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:37:58.743628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58938 (local address 127.10.98.61:35385)
0926 12:37:58.743776 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.743779 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.743852 (+ 73us) server_negotiation.cc:407] Connection header received
0926 12:37:58.743995 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.743997 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.744040 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.744114 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.744614 (+ 500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58938: BlockingRecv error: recv got EOF from 127.0.0.1:58938 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260926 12:37:58.756507 582 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:37:58.755313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58948 (local address 127.10.98.61:35385)
0926 12:37:58.755428 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.755432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.755617 (+ 185us) server_negotiation.cc:407] Connection header received
0926 12:37:58.755768 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.755770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.755820 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.755887 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.756380 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58948: BlockingRecv error: recv got EOF from 127.0.0.1:58948 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 12:37:58.775760 582 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:37:58.774637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58958 (local address 127.10.98.61:35385)
0926 12:37:58.774776 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.774781 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.774892 (+ 111us) server_negotiation.cc:407] Connection header received
0926 12:37:58.775024 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.775027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.775091 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.775188 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.775641 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58958: BlockingRecv error: recv got EOF from 127.0.0.1:58958 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 12:37:58.810883 582 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:37:58.809818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58972 (local address 127.10.98.61:35385)
0926 12:37:58.809962 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.809967 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.810074 (+ 107us) server_negotiation.cc:407] Connection header received
0926 12:37:58.810216 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.810220 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.810265 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.810333 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.810766 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58972: BlockingRecv error: recv got EOF from 127.0.0.1:58972 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260926 12:37:58.878258 582 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:37:58.877079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58980 (local address 127.10.98.61:35385)
0926 12:37:58.877244 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:58.877249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:58.877354 (+ 105us) server_negotiation.cc:407] Connection header received
0926 12:37:58.877509 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:58.877513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:58.877572 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:58.877649 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:58.878137 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58980: BlockingRecv error: recv got EOF from 127.0.0.1:58980 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260926 12:37:59.009635 582 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:37:59.008452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.10.98.61:35385)
0926 12:37:59.008610 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:59.008614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:59.008738 (+ 124us) server_negotiation.cc:407] Connection header received
0926 12:37:59.008899 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:59.008902 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:59.008950 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:59.009014 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:59.009517 (+ 503us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58986: BlockingRecv error: recv got EOF from 127.0.0.1:58986 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260926 12:37:59.269009 582 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:37:59.268024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58996 (local address 127.10.98.61:35385)
0926 12:37:59.268189 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:59.268193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:59.268209 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:37:59.268337 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:59.268339 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:59.268385 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:59.268455 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:37:59.268888 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58996: BlockingRecv error: recv got EOF from 127.0.0.1:58996 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 12:37:59.534747 679 catalog_manager.cc:1259] Loaded cluster ID: 08abc4edac0f40d9b4ff5f10ecfe8f78
I20260926 12:37:59.534832 679 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c3ec323bdcdb480280dcc17ce967d680: loading cluster ID for follower catalog manager: success
I20260926 12:37:59.536542 679 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c3ec323bdcdb480280dcc17ce967d680: acquiring CA information for follower catalog manager: success
I20260926 12:37:59.537743 679 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c3ec323bdcdb480280dcc17ce967d680: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 12:37:59.581434 714 catalog_manager.cc:1259] Loaded cluster ID: 08abc4edac0f40d9b4ff5f10ecfe8f78
I20260926 12:37:59.581514 714 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2deb20352a3549709947af6355ccd4c5: loading cluster ID for follower catalog manager: success
I20260926 12:37:59.583204 714 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2deb20352a3549709947af6355ccd4c5: acquiring CA information for follower catalog manager: success
I20260926 12:37:59.584348 714 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2deb20352a3549709947af6355ccd4c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 12:37:59.787415 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:59.783274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59010 (local address 127.10.98.61:35385)
0926 12:37:59.783612 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:59.783617 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:59.783676 (+ 59us) server_negotiation.cc:407] Connection header received
0926 12:37:59.783835 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:59.783839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:59.783980 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:59.784078 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:37:59.784357 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:59.784891 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:59.785851 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:59.786133 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:59.786238 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:59.786576 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:37:59.787016 (+ 440us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:37:59.787128 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:59.787198 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 12:37:59.787256 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":86,"threads_started":1}
I20260926 12:37:59.794426 793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:59.790706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53324 (local address 127.10.98.60:45767)
0926 12:37:59.790963 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:59.790967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:59.790978 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:37:59.791057 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:59.791059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:59.791175 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:59.791279 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:37:59.791506 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:59.792035 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:59.792859 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:59.793086 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:59.793197 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:59.793490 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:37:59.794011 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:37:59.794124 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:59.794215 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 12:37:59.794273 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":187,"thread_start_us":85,"threads_started":1}
I20260926 12:37:59.801397 799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:59.797283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53436 (local address 127.10.98.62:35357)
0926 12:37:59.797675 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:59.797679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:59.797693 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:37:59.797786 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:59.797789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:59.797920 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:59.798020 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:37:59.798252 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:59.798976 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:59.799804 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:59.800169 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:59.800271 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:59.800614 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:37:59.800945 (+ 331us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:37:59.801063 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:59.801173 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 12:37:59.801237 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":74,"threads_started":1}
I20260926 12:37:59.801549 787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:59.797508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59016 (local address 127.10.98.61:35385)
0926 12:37:59.797631 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:59.797635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:59.798085 (+ 450us) server_negotiation.cc:407] Connection header received
0926 12:37:59.798180 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:59.798183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:59.798236 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:59.798309 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:37:59.798618 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:59.799470 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:59.800326 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:59.800651 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:59.800722 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:59.801070 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:37:59.801174 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:37:59.801234 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:59.801381 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 12:37:59.801437 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:37:59.801868 793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:37:59.797676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53328 (local address 127.10.98.60:45767)
0926 12:37:59.797792 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 12:37:59.797796 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:37:59.798307 (+ 511us) server_negotiation.cc:407] Connection header received
0926 12:37:59.798465 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:37:59.798469 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:37:59.798629 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:37:59.798719 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:37:59.798921 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:59.799638 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:59.800710 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:37:59.801024 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:37:59.801133 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:37:59.801445 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:37:59.801516 (+ 71us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:37:59.801562 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 12:37:59.801719 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 12:37:59.801773 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:37:59.811795 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 430
I20260926 12:37:59.817531 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 508
I20260926 12:37:59.822721 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 584
2026-09-26T12:37:59Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3072 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 12:37:59 dist-test-slave-t7tk krb5kdc[804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 12:37:59 dist-test-slave-t7tk krb5kdc[804](info): set up 2 sockets
Sep 26 12:37:59 dist-test-slave-t7tk krb5kdc[804](info): commencing operation
krb5kdc: starting...
W20260926 12:38:00.779074 10632 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/test-user.keytab.
Sep 26 12:38:00 dist-test-slave-t7tk krb5kdc[804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T12:38:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T12:38:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.10.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
Entry for principal kudu/127.10.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab.
I20260926 12:38:00.901633 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:43317
--webserver_interface=127.10.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:43317,127.10.98.61:36301,127.10.98.60:33279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 12:38:01.014416 817 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 12:38:01.014721 817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:01.014760 817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:01.014823 817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 12:38:01.018975 817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:38:01.019047 817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:01.019081 817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:01.019105 817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:38:01.019134 817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:38:01.024611 817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:43317,127.10.98.61:36301,127.10.98.60:33279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.62.keytab
--principal=kudu/127.10.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.62:43317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.10.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:01.025913 817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:01.026973 817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:38:01.035806 817 server_base.cc:1034] running on GCE node
W20260926 12:38:01.035751 822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:01.035761 823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:01.035751 825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:01.036585 817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:01.037761 817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:01.038923 817 hybrid_clock.cc:648] HybridClock initialized: now 1790426281038903 us; error 34 us; skew 500 ppm
Sep 26 12:38:01 dist-test-slave-t7tk krb5kdc[804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426281, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:01.041762 817 init.cc:377] Logged in from keytab as kudu/127.10.98.62@KRBTEST.COM (short username kudu)
I20260926 12:38:01.042675 817 webserver.cc:460] Webserver started at http://127.10.98.62:43097/ using document root <none> and password file <none>
I20260926 12:38:01.043226 817 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:01.043286 817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:01.043462 817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:01.045576 817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-0/data/instance:
uuid: "3f8aa8d94fe249ffa96d0a36f5387e52"
format_stamp: "Formatted at 2026-09-26 12:38:01 on dist-test-slave-t7tk"
server_key: "c7e195898851c02f11d047d4f71d4847"
server_key_iv: "7059dcd0d656301053b5c45dd44c3e14"
server_key_version: "encryptionkey@0"
I20260926 12:38:01.046106 817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-0/wal/instance:
uuid: "3f8aa8d94fe249ffa96d0a36f5387e52"
format_stamp: "Formatted at 2026-09-26 12:38:01 on dist-test-slave-t7tk"
server_key: "c7e195898851c02f11d047d4f71d4847"
server_key_iv: "7059dcd0d656301053b5c45dd44c3e14"
server_key_version: "encryptionkey@0"
I20260926 12:38:01.049899 817 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260926 12:38:01.052428 832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:01.053856 817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260926 12:38:01.053972 817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-0/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-0/wal
uuid: "3f8aa8d94fe249ffa96d0a36f5387e52"
format_stamp: "Formatted at 2026-09-26 12:38:01 on dist-test-slave-t7tk"
server_key: "c7e195898851c02f11d047d4f71d4847"
server_key_iv: "7059dcd0d656301053b5c45dd44c3e14"
server_key_version: "encryptionkey@0"
I20260926 12:38:01.054085 817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:01.069242 817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:01.072474 817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:01.072664 817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:01.079376 817 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.62:43317
I20260926 12:38:01.079648 885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.62:43317 every 8 connection(s)
I20260926 12:38:01.080549 817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-0/data/info.pb
I20260926 12:38:01.083513 886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:01.087396 886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43317 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:38:01.088183 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 817
I20260926 12:38:01.088312 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-0/wal/instance
I20260926 12:38:01.088654 10632 external_mini_cluster.cc:1428] Setting key edcbbfa3a27bea053bfa6dfedd37626d
WARNING: no policy specified for kudu/127.10.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.61@KRBTEST.COM" created.
Sep 26 12:38:01 dist-test-slave-t7tk krb5kdc[804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426281, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:38:01.106945 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.089898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:43317 (local address 127.0.0.1:36160)
0926 12:38:01.090443 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.090500 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:01.091418 (+ 918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:01.092252 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:01.092258 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:01.092754 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:01.093196 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:01.093220 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.094447 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.094451 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:01.095640 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:01.095652 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.096384 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.096389 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:01.096519 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.097174 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:01.097197 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:01.101317 (+ 4120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:01.103605 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:01.103616 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:01.103629 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:01.103965 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:01.104440 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:01.104443 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:01.104445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:01.104597 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:01.105337 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:01.105345 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:01.105528 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 12:38:01.105737 (+ 209us) client_negotiation.cc:240] Negotiation successful
0926 12:38:01.106182 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":101,"thread_start_us":142,"threads_started":1}
I20260926 12:38:01.106942 890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.089899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36160 (local address 127.10.98.62:43317)
0926 12:38:01.090492 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.090514 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.090637 (+ 123us) server_negotiation.cc:407] Connection header received
0926 12:38:01.091789 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.091795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.092040 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.092209 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.093365 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.094312 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.095778 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.096200 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.096330 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.101510 (+ 5180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:01.101518 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:01.101536 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:01.101567 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:01.103464 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.104102 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.104108 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.104109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.104331 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.104715 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.104718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.104719 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.105222 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:01.105338 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:01.105807 (+ 469us) server_negotiation.cc:299] Negotiation successful
0926 12:38:01.106059 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"spinlock_wait_cycles":58112,"thread_start_us":170,"threads_started":1}
I20260926 12:38:01.109544 886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:38:01.110303 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.110040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:36301 (local address 127.0.0.1:53920)
0926 12:38:01.110167 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.110238 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.61:36301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 12:38:01.110538 836 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.98.61:36301: connect: Connection refused (error 111)
I20260926 12:38:01.111528 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.111334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:36301 (local address 127.0.0.1:53922)
0926 12:38:01.111432 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.111473 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.61:36301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 12:38:01.111698 886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.61:36301: Network error: Client connection negotiation failed: client connection to 127.10.98.61:36301: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.61.keytab.
Entry for principal kudu/127.10.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.61.keytab.
I20260926 12:38:01.117596 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.61:36301
--webserver_interface=127.10.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.61.keytab
--principal=kudu/127.10.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:43317,127.10.98.61:36301,127.10.98.60:33279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:38:01.160974 886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } attempt: 1
I20260926 12:38:01.161981 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.161627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:36301 (local address 127.0.0.1:53930)
0926 12:38:01.161802 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.161883 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.61:36301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 12:38:01.162766 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.162534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:36301 (local address 127.0.0.1:53944)
0926 12:38:01.162643 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.162700 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.61:36301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 12:38:01.162917 886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.61:36301: Network error: Client connection negotiation failed: client connection to 127.10.98.61:36301: connect: Connection refused (error 111)
W20260926 12:38:01.230548 893 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 12:38:01.230863 893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:01.230886 893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:01.230926 893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 12:38:01.231109 886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } attempt: 2
I20260926 12:38:01.231994 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.231669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:36301 (local address 127.0.0.1:53956)
0926 12:38:01.231837 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.231906 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.61:36301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260926 12:38:01.232760 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.232519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:36301 (local address 127.0.0.1:53962)
0926 12:38:01.232643 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.232699 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.61:36301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 12:38:01.232892 886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.61:36301: Network error: Client connection negotiation failed: client connection to 127.10.98.61:36301: connect: Connection refused (error 111)
W20260926 12:38:01.235461 893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:38:01.235518 893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:01.235536 893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:01.235553 893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:38:01.235577 893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:38:01.241009 893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:43317,127.10.98.61:36301,127.10.98.60:33279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.61.keytab
--principal=kudu/127.10.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.61:36301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.10.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.893
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:01.242358 893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:01.243614 893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:38:01.252499 893 server_base.cc:1034] running on GCE node
W20260926 12:38:01.252476 898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:01.252564 899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:01.252490 901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:01.253371 893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:01.254590 893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:01.255754 893 hybrid_clock.cc:648] HybridClock initialized: now 1790426281255736 us; error 37 us; skew 500 ppm
Sep 26 12:38:01 dist-test-slave-t7tk krb5kdc[804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426281, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:01.258566 893 init.cc:377] Logged in from keytab as kudu/127.10.98.61@KRBTEST.COM (short username kudu)
I20260926 12:38:01.259490 893 webserver.cc:460] Webserver started at http://127.10.98.61:39989/ using document root <none> and password file <none>
I20260926 12:38:01.260000 893 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:01.260057 893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:01.260241 893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:01.262303 893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-1/data/instance:
uuid: "945659a260fb4f03bfe4fff2d9cd32e8"
format_stamp: "Formatted at 2026-09-26 12:38:01 on dist-test-slave-t7tk"
server_key: "0dcbde284f03aacfee8fcf3e381e983e"
server_key_iv: "df3c5b139759320a37511e80f42c0398"
server_key_version: "encryptionkey@0"
I20260926 12:38:01.262821 893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-1/wal/instance:
uuid: "945659a260fb4f03bfe4fff2d9cd32e8"
format_stamp: "Formatted at 2026-09-26 12:38:01 on dist-test-slave-t7tk"
server_key: "0dcbde284f03aacfee8fcf3e381e983e"
server_key_iv: "df3c5b139759320a37511e80f42c0398"
server_key_version: "encryptionkey@0"
I20260926 12:38:01.266502 893 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260926 12:38:01.268949 908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:01.270253 893 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260926 12:38:01.270364 893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-1/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-1/wal
uuid: "945659a260fb4f03bfe4fff2d9cd32e8"
format_stamp: "Formatted at 2026-09-26 12:38:01 on dist-test-slave-t7tk"
server_key: "0dcbde284f03aacfee8fcf3e381e983e"
server_key_iv: "df3c5b139759320a37511e80f42c0398"
server_key_version: "encryptionkey@0"
I20260926 12:38:01.270471 893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:01.286514 893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:01.289764 893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:01.289945 893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:01.296861 893 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.61:36301
I20260926 12:38:01.297142 960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.61:36301 every 8 connection(s)
I20260926 12:38:01.297888 893 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-1/data/info.pb
I20260926 12:38:01.300751 961 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:01.304598 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 893
I20260926 12:38:01.304731 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-1/wal/instance
I20260926 12:38:01.305020 10632 external_mini_cluster.cc:1428] Setting key 27e1f402652980e5c4a5e5141234b214
I20260926 12:38:01.304772 961 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43317 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.10.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.10.98.60@KRBTEST.COM" created.
Sep 26 12:38:01 dist-test-slave-t7tk krb5kdc[804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426281, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.61@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:38:01.320117 890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.307145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36168 (local address 127.10.98.62:43317)
0926 12:38:01.307273 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.307277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.308376 (+ 1099us) server_negotiation.cc:407] Connection header received
0926 12:38:01.309456 (+ 1080us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.309460 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.309520 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.309594 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.310780 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.311297 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.312418 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.312698 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.312791 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.316336 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:01.316340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:01.316343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:01.316376 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:01.317830 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.318323 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.318325 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.318327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.318393 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.318886 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.318890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.318892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.319107 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:01.319192 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:01.319861 (+ 669us) server_negotiation.cc:299] Negotiation successful
0926 12:38:01.319982 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 12:38:01.321439 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.307678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:43317 (local address 127.0.0.1:36168)
0926 12:38:01.308208 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.308253 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:01.309254 (+ 1001us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:01.309776 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:01.309782 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:01.310156 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:01.310637 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:01.310650 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.311419 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.311422 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:01.312297 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:01.312306 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.312845 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.312849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:01.313015 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.313593 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:01.313610 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:01.316141 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:01.317955 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:01.317962 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:01.317978 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:01.318243 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:01.318562 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:01.318566 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:01.318568 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:01.318762 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:01.319208 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:01.319214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:01.319553 (+ 339us) client_negotiation.cc:769] Sending connection context
0926 12:38:01.319784 (+ 231us) client_negotiation.cc:240] Negotiation successful
0926 12:38:01.320164 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":149,"threads_started":1}
I20260926 12:38:01.323097 961 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:38:01.324074 886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } attempt: 3
Sep 26 12:38:01 dist-test-slave-t7tk krb5kdc[804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426281, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.61@KRBTEST.COM for kudu/127.10.98.61@KRBTEST.COM
Sep 26 12:38:01 dist-test-slave-t7tk krb5kdc[804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426281, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for kudu/127.10.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.10.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.60.keytab.
Entry for principal kudu/127.10.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.60.keytab.
I20260926 12:38:01.332734 10632 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
/tmp/dist-test-taskFMA2CE/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.60:33279
--webserver_interface=127.10.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.60.keytab
--principal=kudu/127.10.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.10.98.20:32817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.10.98.62:43317,127.10.98.61:36301,127.10.98.60:33279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 12:38:01.333468 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.323683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:36301 (local address 127.0.0.1:53974)
0926 12:38:01.323830 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.323845 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:01.323931 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:01.324748 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:01.324750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:01.324891 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:01.324978 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:01.324984 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.326024 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.326027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:01.326690 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:01.326700 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.327266 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.327269 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:01.327441 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.327869 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:01.327883 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:01.329377 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:01.331690 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:01.331693 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:01.331695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:01.332127 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:01.332392 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:01.332395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:01.332397 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:01.332465 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:01.332966 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:01.332968 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:01.333043 (+ 75us) client_negotiation.cc:769] Sending connection context
0926 12:38:01.333164 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 12:38:01.333297 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":53}
I20260926 12:38:01.333774 966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.323813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53974 (local address 127.10.98.61:36301)
0926 12:38:01.324132 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.324142 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.324161 (+ 19us) server_negotiation.cc:407] Connection header received
0926 12:38:01.324205 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.324210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.324446 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.324595 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.325083 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.325919 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.326795 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.327165 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.327253 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.329520 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:01.329531 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:01.329548 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:01.329570 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:01.331587 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.332229 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.332235 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.332237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.332314 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.332539 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.332542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.332544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.332804 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:01.332935 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:01.333267 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 12:38:01.333392 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":225,"spinlock_wait_cycles":1408,"thread_start_us":111,"threads_started":1}
I20260926 12:38:01.333837 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.324449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:36301 (local address 127.0.0.1:53984)
0926 12:38:01.324637 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.324653 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:01.324741 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:01.325247 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:01.325250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:01.325472 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:01.325596 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:01.325602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.326452 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.326456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:01.327529 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:01.327540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.328039 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.328042 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:01.328124 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.328555 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:01.328568 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:01.329917 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:01.332246 (+ 2329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:01.332249 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:01.332252 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:01.332490 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:01.332828 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:01.332831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:01.332833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:01.332921 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:01.333384 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:01.333387 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:01.333482 (+ 95us) client_negotiation.cc:769] Sending connection context
0926 12:38:01.333556 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 12:38:01.333680 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 12:38:01.334421 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.324606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53984 (local address 127.10.98.61:36301)
0926 12:38:01.325016 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.325022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.325032 (+ 10us) server_negotiation.cc:407] Connection header received
0926 12:38:01.325060 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.325062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.325092 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.325200 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.325699 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.326345 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.327649 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.327925 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.328041 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.330113 (+ 2072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:01.330117 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:01.330120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:01.330144 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:01.332121 (+ 1977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.332613 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.332616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.332618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.332719 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.333035 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.333038 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.333039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.333281 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:01.334111 (+ 830us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:01.334190 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 12:38:01.334301 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"spinlock_wait_cycles":3456,"thread_start_us":132,"threads_started":1}
I20260926 12:38:01.336170 886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:38:01.336365 961 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:38:01.336849 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.336611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:33279 (local address 127.0.0.1:45548)
0926 12:38:01.336726 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.336782 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 12:38:01.337090 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.336748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:33279 (local address 127.0.0.1:45550)
0926 12:38:01.336835 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.336903 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 12:38:01.337507 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.337270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:33279 (local address 127.0.0.1:45562)
0926 12:38:01.337396 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.337446 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260926 12:38:01.337512 911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.10.98.60:33279: connect: Connection refused (error 111)
W20260926 12:38:01.338220 886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.60:33279: Network error: Client connection negotiation failed: client connection to 127.10.98.60:33279: connect: Connection refused (error 111)
I20260926 12:38:01.338263 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.338075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:33279 (local address 127.0.0.1:45568)
0926 12:38:01.338164 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.338208 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 12:38:01.338385 961 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.60:33279: Network error: Client connection negotiation failed: client connection to 127.10.98.60:33279: connect: Connection refused (error 111)
I20260926 12:38:01.369393 886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } attempt: 1
I20260926 12:38:01.370304 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.369989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:33279 (local address 127.0.0.1:45580)
0926 12:38:01.370156 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.370224 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 12:38:01.371083 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.370843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:33279 (local address 127.0.0.1:45588)
0926 12:38:01.370962 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.371014 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 12:38:01.371222 886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.60:33279: Network error: Client connection negotiation failed: client connection to 127.10.98.60:33279: connect: Connection refused (error 111)
I20260926 12:38:01.387655 961 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } attempt: 1
I20260926 12:38:01.388641 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.388320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:33279 (local address 127.0.0.1:45602)
0926 12:38:01.388486 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.388558 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 12:38:01.389395 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.389194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:33279 (local address 127.0.0.1:45612)
0926 12:38:01.389292 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.389339 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 12:38:01.389544 961 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.60:33279: Network error: Client connection negotiation failed: client connection to 127.10.98.60:33279: connect: Connection refused (error 111)
I20260926 12:38:01.446386 886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } attempt: 2
I20260926 12:38:01.447225 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.446931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:33279 (local address 127.0.0.1:45622)
0926 12:38:01.447087 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.447148 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 12:38:01.447383 968 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 12:38:01.447773 968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 12:38:01.447820 968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 12:38:01.447885 968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260926 12:38:01.447956 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.447727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:33279 (local address 127.0.0.1:45634)
0926 12:38:01.447839 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.447888 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260926 12:38:01.448060 886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.60:33279: Network error: Client connection negotiation failed: client connection to 127.10.98.60:33279: connect: Connection refused (error 111)
W20260926 12:38:01.452076 968 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 12:38:01.452140 968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 12:38:01.452165 968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 12:38:01.452188 968 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 12:38:01.452216 968 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 12:38:01.457731 968 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.10.98.20:32817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.10.98.62:43317,127.10.98.61:36301,127.10.98.60:33279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/krb5kdc/kudu_127.10.98.60.keytab
--principal=kudu/127.10.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.10.98.60:33279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.10.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.968
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 26 Sep 2026 11:57:44 UTC on ccycloud.quasar-qryufi.root.comops.site
build id 4674111
I20260926 12:38:01.457713 961 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } attempt: 2
I20260926 12:38:01.458575 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.458210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:33279 (local address 127.0.0.1:45644)
0926 12:38:01.458395 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.458486 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260926 12:38:01.459025 968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 12:38:01.459333 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.459108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:33279 (local address 127.0.0.1:45660)
0926 12:38:01.459215 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.459269 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.10.98.60:33279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 12:38:01.459517 961 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.10.98.60:33279: Network error: Client connection negotiation failed: client connection to 127.10.98.60:33279: connect: Connection refused (error 111)
I20260926 12:38:01.460180 968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 12:38:01.468997 968 server_base.cc:1034] running on GCE node
W20260926 12:38:01.468921 974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:01.468921 977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 12:38:01.468982 975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 12:38:01.469899 968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 12:38:01.471205 968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 12:38:01.472406 968 hybrid_clock.cc:648] HybridClock initialized: now 1790426281472386 us; error 32 us; skew 500 ppm
Sep 26 12:38:01 dist-test-slave-t7tk krb5kdc[804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426281, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 12:38:01.475389 968 init.cc:377] Logged in from keytab as kudu/127.10.98.60@KRBTEST.COM (short username kudu)
I20260926 12:38:01.476352 968 webserver.cc:460] Webserver started at http://127.10.98.60:42851/ using document root <none> and password file <none>
I20260926 12:38:01.476897 968 fs_manager.cc:359] Metadata directory not provided
I20260926 12:38:01.476958 968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 12:38:01.477180 968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 12:38:01.479260 968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-2/data/instance:
uuid: "8b1d107b715a4bb7b66abee1e3ce605a"
format_stamp: "Formatted at 2026-09-26 12:38:01 on dist-test-slave-t7tk"
server_key: "4c7824bc1d8407ec54e0a6f2b89bf4ad"
server_key_iv: "87d05ddd6715004d465cd84351ea8ce1"
server_key_version: "encryptionkey@0"
I20260926 12:38:01.479784 968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-2/wal/instance:
uuid: "8b1d107b715a4bb7b66abee1e3ce605a"
format_stamp: "Formatted at 2026-09-26 12:38:01 on dist-test-slave-t7tk"
server_key: "4c7824bc1d8407ec54e0a6f2b89bf4ad"
server_key_iv: "87d05ddd6715004d465cd84351ea8ce1"
server_key_version: "encryptionkey@0"
I20260926 12:38:01.483767 968 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260926 12:38:01.486471 984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 12:38:01.487867 968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 12:38:01.488022 968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-2/data,/tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-2/wal
uuid: "8b1d107b715a4bb7b66abee1e3ce605a"
format_stamp: "Formatted at 2026-09-26 12:38:01 on dist-test-slave-t7tk"
server_key: "4c7824bc1d8407ec54e0a6f2b89bf4ad"
server_key_iv: "87d05ddd6715004d465cd84351ea8ce1"
server_key_version: "encryptionkey@0"
I20260926 12:38:01.488147 968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 12:38:01.502051 968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 12:38:01.505183 968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 12:38:01.505369 968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 12:38:01.512354 968 rpc_server.cc:307] RPC server started. Bound to: 127.10.98.60:33279
I20260926 12:38:01.512630 1037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.10.98.60:33279 every 8 connection(s)
I20260926 12:38:01.513391 968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-2/data/info.pb
I20260926 12:38:01.516248 1038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 12:38:01.520191 1038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43317 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:38:01.520629 10632 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu as pid 968
I20260926 12:38:01.520757 10632 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskFMA2CE/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790425997379116-10632-0/minicluster-data/master-2/wal/instance
I20260926 12:38:01.521040 10632 external_mini_cluster.cc:1428] Setting key 66520e9637ae2dc67eca8cd892b1de87
Sep 26 12:38:01 dist-test-slave-t7tk krb5kdc[804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426281, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.60@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
Sep 26 12:38:01 dist-test-slave-t7tk krb5kdc[804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.60@KRBTEST.COM
I20260926 12:38:01.534915 890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.522325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36174 (local address 127.10.98.62:43317)
0926 12:38:01.522499 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.522504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.523113 (+ 609us) server_negotiation.cc:407] Connection header received
0926 12:38:01.523977 (+ 864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.523980 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.524029 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.524108 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.525254 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.526102 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.527499 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.527916 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.528031 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.530914 (+ 2883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:01.530918 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:01.530920 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:01.530948 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:01.532410 (+ 1462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.533078 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.533082 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.533083 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.533196 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.533649 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.533653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.533655 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.533882 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:01.534013 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:01.534616 (+ 603us) server_negotiation.cc:299] Negotiation successful
0926 12:38:01.534747 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 12:38:01.535844 1041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.522494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.62:43317 (local address 127.0.0.1:36174)
0926 12:38:01.522953 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.523003 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:01.523745 (+ 742us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:01.524207 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:01.524213 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:01.524669 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:01.525089 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:01.525135 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.526236 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.526238 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:01.527357 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:01.527368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.528099 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.528102 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:01.528242 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.528905 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:01.528927 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:01.530775 (+ 1848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:01.532592 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:01.532599 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:01.532619 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:01.532977 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:01.533281 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:01.533284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:01.533286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:01.533535 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:01.533993 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:01.533999 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:01.534288 (+ 289us) client_negotiation.cc:769] Sending connection context
0926 12:38:01.534546 (+ 258us) client_negotiation.cc:240] Negotiation successful
0926 12:38:01.535032 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":184,"thread_start_us":111,"threads_started":1}
I20260926 12:38:01.535848 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.522493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45672 (local address 127.10.98.60:33279)
0926 12:38:01.522931 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.522949 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.522981 (+ 32us) server_negotiation.cc:407] Connection header received
0926 12:38:01.523702 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.523721 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.523947 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.524143 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.525179 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.526379 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.527581 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.528186 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.528326 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.531268 (+ 2942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:01.531274 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:01.531285 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:01.531310 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:01.532969 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.533440 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.533445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.533446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.533611 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.533910 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.533914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.533915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.534280 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:01.534420 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:01.534696 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 12:38:01.534910 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":313,"spinlock_wait_cycles":21248,"thread_start_us":157,"threads_started":1}
I20260926 12:38:01.537176 1038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 12:38:01 dist-test-slave-t7tk krb5kdc[804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426281, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.60@KRBTEST.COM for kudu/127.10.98.61@KRBTEST.COM
Sep 26 12:38:01 dist-test-slave-t7tk krb5kdc[804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.62@KRBTEST.COM
I20260926 12:38:01.545444 1041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.537671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.61:36301 (local address 127.0.0.1:53996)
0926 12:38:01.537786 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.537801 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:01.537871 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:01.538079 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:01.538083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:01.538236 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:01.538350 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:01.538357 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.539281 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.539284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:01.539830 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:01.539839 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.540366 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.540370 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:01.540465 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.540906 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:01.540920 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:01.542137 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:01.543910 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:01.543914 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:01.543915 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:01.544181 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:01.544460 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:01.544464 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:01.544466 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:01.544555 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:01.544937 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:01.544940 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:01.545021 (+ 81us) client_negotiation.cc:769] Sending connection context
0926 12:38:01.545153 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 12:38:01.545300 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 12:38:01.545559 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.537772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53996 (local address 127.10.98.61:36301)
0926 12:38:01.537881 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.537885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.537899 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:38:01.537937 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.537939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.537979 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.538048 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.538578 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.539169 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.539994 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.540262 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.540351 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.542258 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:01.542261 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:01.542263 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:01.542288 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:01.543789 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.544275 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.544287 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.544288 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.544372 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.544637 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.544639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.544640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.544849 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:01.544946 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:01.545203 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 12:38:01.545321 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 12:38:01.546335 890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.538397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36188 (local address 127.10.98.62:43317)
0926 12:38:01.538506 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.538510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.538524 (+ 14us) server_negotiation.cc:407] Connection header received
0926 12:38:01.538591 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.538594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.538637 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.538709 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.539226 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.539717 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.540528 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.540734 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.540823 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.543072 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:01.543076 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:01.543078 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:01.543114 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:01.544647 (+ 1533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.545086 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.545089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.545091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.545220 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.545486 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.545489 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.545490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.545727 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:01.545943 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:01.546099 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 12:38:01.546201 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 12:38:01.546641 1038 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } has no permanent_uuid. Determining permanent_uuid...
I20260926 12:38:01.547259 886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } attempt: 3
I20260926 12:38:01.550685 961 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } attempt: 3
Sep 26 12:38:01 dist-test-slave-t7tk krb5kdc[804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426281, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.60@KRBTEST.COM for kudu/127.10.98.60@KRBTEST.COM
Sep 26 12:38:01 dist-test-slave-t7tk krb5kdc[804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426281, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.62@KRBTEST.COM for kudu/127.10.98.60@KRBTEST.COM
Sep 26 12:38:01 dist-test-slave-t7tk krb5kdc[804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426280, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.10.98.61@KRBTEST.COM
Sep 26 12:38:01 dist-test-slave-t7tk krb5kdc[804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790426281, etypes {rep=17 tkt=17 ses=17}, kudu/127.10.98.61@KRBTEST.COM for kudu/127.10.98.60@KRBTEST.COM
I20260926 12:38:01.558606 889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.547580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:33279 (local address 127.0.0.1:45686)
0926 12:38:01.547686 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.547699 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:01.547770 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:01.548208 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:01.548211 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:01.548371 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:01.548448 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:01.548453 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.549741 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.549744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:01.550538 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:01.550548 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.551112 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.551116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:01.551197 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.551735 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:01.551748 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:01.553126 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:01.555544 (+ 2418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:01.555548 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:01.555550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:01.555786 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:01.557115 (+ 1329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:01.557118 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:01.557120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:01.557217 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:01.558102 (+ 885us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:01.558105 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:01.558210 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 12:38:01.558291 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 12:38:01.558431 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 12:38:01.558884 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.547570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53998 (local address 127.10.98.61:36301)
0926 12:38:01.547675 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.547679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.547804 (+ 125us) server_negotiation.cc:407] Connection header received
0926 12:38:01.547972 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.547975 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.548017 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.548066 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.548864 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.549527 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.550573 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.550878 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.551001 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.554345 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:01.554349 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:01.554352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:01.554379 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:01.555938 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.556405 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.556409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.556410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.556515 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.556801 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.556805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.556806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.557023 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:01.557285 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:01.558621 (+ 1336us) server_negotiation.cc:299] Negotiation successful
0926 12:38:01.558735 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 12:38:01.559165 1043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.547700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45686 (local address 127.10.98.60:33279)
0926 12:38:01.548021 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.548028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.548037 (+ 9us) server_negotiation.cc:407] Connection header received
0926 12:38:01.548062 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.548071 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.548102 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.548172 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.548545 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.549630 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.550658 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.550994 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.551399 (+ 405us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.553275 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:01.553279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:01.553282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:01.553308 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:01.555121 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.556897 (+ 1776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.556899 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.556901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.556981 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.557332 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.557336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.557337 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.557598 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:01.558786 (+ 1188us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:01.558862 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 12:38:01.558981 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":271,"thread_start_us":133,"threads_started":1}
I20260926 12:38:01.560101 1041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.547132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:33279 (local address 127.0.0.1:45674)
0926 12:38:01.547239 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.547253 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:01.547323 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:01.548014 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:01.548016 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:01.548178 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:01.548317 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:01.548324 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.549299 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.549302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:01.549957 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:01.549963 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.550364 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.550367 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:01.550432 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.550971 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:01.550986 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:01.553763 (+ 2777us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:01.556565 (+ 2802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:01.556568 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:01.556570 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:01.556785 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:01.557859 (+ 1074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:01.557862 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:01.557863 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:01.557939 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:01.559626 (+ 1687us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:01.559628 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:01.559707 (+ 79us) client_negotiation.cc:769] Sending connection context
0926 12:38:01.559800 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 12:38:01.559943 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":26}
I20260926 12:38:01.560532 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.547204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45674 (local address 127.10.98.60:33279)
0926 12:38:01.547759 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.547763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.547774 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:38:01.547807 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.547809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.547850 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.547918 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.548426 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.549185 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.550042 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.550277 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.550358 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.554243 (+ 3885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:01.554247 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:01.554250 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:01.554275 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:01.556458 (+ 2183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.557693 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.557696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.557698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.557770 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.559303 (+ 1533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.559306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.559308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.559500 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:01.559591 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:01.560278 (+ 687us) server_negotiation.cc:299] Negotiation successful
0926 12:38:01.560383 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488}
I20260926 12:38:01.562532 10632 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 12:38:01.562702 964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.551064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.10.98.60:33279 (local address 127.0.0.1:45700)
0926 12:38:01.551300 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 12:38:01.551315 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 12:38:01.551982 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 12:38:01.552494 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 12:38:01.552497 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 12:38:01.552695 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 12:38:01.552802 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:01.552809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.554434 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.554438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:01.555989 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 12:38:01.555997 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.556562 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.556566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 12:38:01.556645 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.557066 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 12:38:01.557079 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 12:38:01.558483 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 12:38:01.561070 (+ 2587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:01.561073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:01.561075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:01.561363 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:01.561681 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 12:38:01.561684 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 12:38:01.561686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 12:38:01.561778 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 12:38:01.562190 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 12:38:01.562193 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 12:38:01.562287 (+ 94us) client_negotiation.cc:769] Sending connection context
0926 12:38:01.562366 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 12:38:01.562519 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165}
I20260926 12:38:01.563194 1044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.551771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45700 (local address 127.10.98.60:33279)
0926 12:38:01.552203 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.552211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.552222 (+ 11us) server_negotiation.cc:407] Connection header received
0926 12:38:01.552255 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.552257 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.552299 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.552914 (+ 615us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.553055 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.554116 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.556110 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.556446 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.557588 (+ 1142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.558621 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:01.558625 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:01.558627 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:01.558648 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:01.560953 (+ 2305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.561483 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.561487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.561489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.561573 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.561885 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.561887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.561889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.562082 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:01.562856 (+ 774us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:01.562926 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 12:38:01.563027 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":98,"threads_started":1}
I20260926 12:38:01.565308 1038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b1d107b715a4bb7b66abee1e3ce605a: Bootstrap starting.
I20260926 12:38:01.565310 886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52: Bootstrap starting.
I20260926 12:38:01.567476 1038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b1d107b715a4bb7b66abee1e3ce605a: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:01.567478 886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:01.568324 1038 log.cc:824] T 00000000000000000000000000000000 P 8b1d107b715a4bb7b66abee1e3ce605a: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:01.569206 886 log.cc:824] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:01.568111 860 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36188
I20260926 12:38:01.570526 1038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b1d107b715a4bb7b66abee1e3ce605a: No bootstrap required, opened a new log
I20260926 12:38:01.570608 961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 945659a260fb4f03bfe4fff2d9cd32e8: Bootstrap starting.
I20260926 12:38:01.571295 886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52: No bootstrap required, opened a new log
I20260926 12:38:01.573344 961 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 945659a260fb4f03bfe4fff2d9cd32e8: Neither blocks nor log segments found. Creating new log.
I20260926 12:38:01.574184 961 log.cc:824] T 00000000000000000000000000000000 P 945659a260fb4f03bfe4fff2d9cd32e8: Log is configured to *not* fsync() on all Append() calls
I20260926 12:38:01.574295 886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43317 } } peers { permanent_uuid: "945659a260fb4f03bfe4fff2d9cd32e8" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } } peers { permanent_uuid: "8b1d107b715a4bb7b66abee1e3ce605a" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } }
I20260926 12:38:01.574541 886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f8aa8d94fe249ffa96d0a36f5387e52, State: Initialized, Role: FOLLOWER
I20260926 12:38:01.574880 886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43317 } } peers { permanent_uuid: "945659a260fb4f03bfe4fff2d9cd32e8" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } } peers { permanent_uuid: "8b1d107b715a4bb7b66abee1e3ce605a" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } }
I20260926 12:38:01.575157 1038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b1d107b715a4bb7b66abee1e3ce605a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43317 } } peers { permanent_uuid: "945659a260fb4f03bfe4fff2d9cd32e8" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } } peers { permanent_uuid: "8b1d107b715a4bb7b66abee1e3ce605a" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } }
I20260926 12:38:01.575397 1038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b1d107b715a4bb7b66abee1e3ce605a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b1d107b715a4bb7b66abee1e3ce605a, State: Initialized, Role: FOLLOWER
I20260926 12:38:01.575580 1048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43317 } } peers { permanent_uuid: "945659a260fb4f03bfe4fff2d9cd32e8" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } } peers { permanent_uuid: "8b1d107b715a4bb7b66abee1e3ce605a" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } } }
I20260926 12:38:01.575707 1038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b1d107b715a4bb7b66abee1e3ce605a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43317 } } peers { permanent_uuid: "945659a260fb4f03bfe4fff2d9cd32e8" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } } peers { permanent_uuid: "8b1d107b715a4bb7b66abee1e3ce605a" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } }
I20260926 12:38:01.575910 1048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:38:01.576210 961 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 945659a260fb4f03bfe4fff2d9cd32e8: No bootstrap required, opened a new log
I20260926 12:38:01.576203 886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:38:01.576267 860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 12:38:01.576366 860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:01.576953 1038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b1d107b715a4bb7b66abee1e3ce605a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:38:01.577798 860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43317 } } peers { permanent_uuid: "945659a260fb4f03bfe4fff2d9cd32e8" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } } peers { permanent_uuid: "8b1d107b715a4bb7b66abee1e3ce605a" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } }
I20260926 12:38:01.578828 860 leader_election.cc:288] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [CANDIDATE]: Term 1 election: Requested vote from peers 945659a260fb4f03bfe4fff2d9cd32e8 (127.10.98.61:36301), 8b1d107b715a4bb7b66abee1e3ce605a (127.10.98.60:33279)
I20260926 12:38:01.580466 1012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8b1d107b715a4bb7b66abee1e3ce605a"
I20260926 12:38:01.580683 1012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b1d107b715a4bb7b66abee1e3ce605a [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:01.580713 961 raft_consensus.cc:348] T 00000000000000000000000000000000 P 945659a260fb4f03bfe4fff2d9cd32e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43317 } } peers { permanent_uuid: "945659a260fb4f03bfe4fff2d9cd32e8" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } } peers { permanent_uuid: "8b1d107b715a4bb7b66abee1e3ce605a" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } }
I20260926 12:38:01.580925 961 raft_consensus.cc:729] T 00000000000000000000000000000000 P 945659a260fb4f03bfe4fff2d9cd32e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 945659a260fb4f03bfe4fff2d9cd32e8, State: Initialized, Role: FOLLOWER
I20260926 12:38:01.581265 961 consensus_queue.cc:260] T 00000000000000000000000000000000 P 945659a260fb4f03bfe4fff2d9cd32e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43317 } } peers { permanent_uuid: "945659a260fb4f03bfe4fff2d9cd32e8" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } } peers { permanent_uuid: "8b1d107b715a4bb7b66abee1e3ce605a" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } }
I20260926 12:38:01.582057 1012 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8b1d107b715a4bb7b66abee1e3ce605a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f8aa8d94fe249ffa96d0a36f5387e52 in term 1.
W20260926 12:38:01.582283 1071 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8b1d107b715a4bb7b66abee1e3ce605a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:38:01.582352 1071 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:38:01.582533 1049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b1d107b715a4bb7b66abee1e3ce605a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43317 } } peers { permanent_uuid: "945659a260fb4f03bfe4fff2d9cd32e8" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } } peers { permanent_uuid: "8b1d107b715a4bb7b66abee1e3ce605a" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } } }
I20260926 12:38:01.582556 835 leader_election.cc:302] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f8aa8d94fe249ffa96d0a36f5387e52, 8b1d107b715a4bb7b66abee1e3ce605a; no voters:
I20260926 12:38:01.582643 1049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b1d107b715a4bb7b66abee1e3ce605a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:38:01.582931 961 sys_catalog.cc:564] T 00000000000000000000000000000000 P 945659a260fb4f03bfe4fff2d9cd32e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 12:38:01.583153 935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "945659a260fb4f03bfe4fff2d9cd32e8"
I20260926 12:38:01.583379 935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 945659a260fb4f03bfe4fff2d9cd32e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 12:38:01.584721 935 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 945659a260fb4f03bfe4fff2d9cd32e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f8aa8d94fe249ffa96d0a36f5387e52 in term 1.
I20260926 12:38:01.587016 1048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 12:38:01.587314 1048 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [term 1 LEADER]: Becoming Leader. State: Replica: 3f8aa8d94fe249ffa96d0a36f5387e52, State: Running, Role: LEADER
I20260926 12:38:01.587618 1077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 945659a260fb4f03bfe4fff2d9cd32e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43317 } } peers { permanent_uuid: "945659a260fb4f03bfe4fff2d9cd32e8" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } } peers { permanent_uuid: "8b1d107b715a4bb7b66abee1e3ce605a" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } } }
I20260926 12:38:01.587759 1077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 945659a260fb4f03bfe4fff2d9cd32e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:38:01.587714 1048 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43317 } } peers { permanent_uuid: "945659a260fb4f03bfe4fff2d9cd32e8" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } } peers { permanent_uuid: "8b1d107b715a4bb7b66abee1e3ce605a" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } }
I20260926 12:38:01.584599 817 master_runner.cc:186] Error getting master registration for 127.10.98.61:36301: OK, instance_id { permanent_uuid: "945659a260fb4f03bfe4fff2d9cd32e8" instance_seqno: 1790426281294930 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 12:38:01.588414 817 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 12:38:01.590001 1048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f8aa8d94fe249ffa96d0a36f5387e52. Latest consensus state: current_term: 1 leader_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43317 } } peers { permanent_uuid: "945659a260fb4f03bfe4fff2d9cd32e8" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } } peers { permanent_uuid: "8b1d107b715a4bb7b66abee1e3ce605a" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } } }
I20260926 12:38:01.590227 1048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:01.591743 968 master_runner.cc:263] No leader master found from master 8b1d107b715a4bb7b66abee1e3ce605a
I20260926 12:38:01.591809 968 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 12:38:01.592384 1088 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 945659a260fb4f03bfe4fff2d9cd32e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 12:38:01.592459 1088 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 12:38:01.594050 893 master_runner.cc:263] No leader master found from master 945659a260fb4f03bfe4fff2d9cd32e8
I20260926 12:38:01.594110 893 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 12:38:01.600380 1044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.590064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45702 (local address 127.10.98.60:33279)
0926 12:38:01.590197 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.590201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.592242 (+ 2041us) server_negotiation.cc:407] Connection header received
0926 12:38:01.592410 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.592413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.592466 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.592514 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.593740 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.594413 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.595249 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.595506 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.595592 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.596898 (+ 1306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:01.596902 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:01.596904 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:01.596930 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:01.598530 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.598987 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.598991 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.598993 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.599092 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.599374 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.599377 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.599379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.599585 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:01.599680 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:01.600024 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 12:38:01.600180 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:38:01.601229 1093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 12:38:01.605718 890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.589502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36196 (local address 127.10.98.62:43317)
0926 12:38:01.589619 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.589624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.591250 (+ 1626us) server_negotiation.cc:407] Connection header received
0926 12:38:01.591426 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.591429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.591477 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.591523 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.597975 (+ 6452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.598733 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.599851 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.600158 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.600235 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.602335 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:01.602339 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:01.602342 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:01.602368 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:01.603898 (+ 1530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.604352 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.604355 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.604357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.604453 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.604732 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.604736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.604737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.604951 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:01.605070 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:01.605420 (+ 350us) server_negotiation.cc:299] Negotiation successful
0926 12:38:01.605555 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 12:38:01.606029 967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:01.590495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.10.98.61:36301)
0926 12:38:01.590603 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.590607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.597352 (+ 6745us) server_negotiation.cc:407] Connection header received
0926 12:38:01.597502 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.597505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.597553 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.597615 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.598410 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.599177 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.600265 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:01.600601 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:01.600675 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:01.602433 (+ 1758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 12:38:01.602436 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 12:38:01.602439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 12:38:01.602464 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 12:38:01.604320 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.604732 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.604736 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.604737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.604812 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 12:38:01.605138 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 12:38:01.605141 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 12:38:01.605143 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0926 12:38:01.605343 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 12:38:01.605443 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:01.605764 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 12:38:01.605870 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 12:38:01.610482 1093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 12:38:01.616674 935 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 945659a260fb4f03bfe4fff2d9cd32e8 [term 1 FOLLOWER]: Refusing update from remote peer 3f8aa8d94fe249ffa96d0a36f5387e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:01.616731 1012 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8b1d107b715a4bb7b66abee1e3ce605a [term 1 FOLLOWER]: Refusing update from remote peer 3f8aa8d94fe249ffa96d0a36f5387e52: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 12:38:01.617383 1089 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "945659a260fb4f03bfe4fff2d9cd32e8" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:01.619342 1089 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8b1d107b715a4bb7b66abee1e3ce605a" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 12:38:01.623052 1077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 945659a260fb4f03bfe4fff2d9cd32e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f8aa8d94fe249ffa96d0a36f5387e52. Latest consensus state: current_term: 1 leader_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43317 } } peers { permanent_uuid: "945659a260fb4f03bfe4fff2d9cd32e8" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } } peers { permanent_uuid: "8b1d107b715a4bb7b66abee1e3ce605a" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } } }
I20260926 12:38:01.623217 1077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 945659a260fb4f03bfe4fff2d9cd32e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:38:01.629371 1049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b1d107b715a4bb7b66abee1e3ce605a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f8aa8d94fe249ffa96d0a36f5387e52. Latest consensus state: current_term: 1 leader_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43317 } } peers { permanent_uuid: "945659a260fb4f03bfe4fff2d9cd32e8" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } } peers { permanent_uuid: "8b1d107b715a4bb7b66abee1e3ce605a" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } } }
I20260926 12:38:01.629547 1049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b1d107b715a4bb7b66abee1e3ce605a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:38:01.634408 1089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43317 } } peers { permanent_uuid: "945659a260fb4f03bfe4fff2d9cd32e8" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } } peers { permanent_uuid: "8b1d107b715a4bb7b66abee1e3ce605a" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } } }
I20260926 12:38:01.634666 1089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:01.634802 1097 mvcc.cc:205] Tried to move back new op lower bound from 7333586049494278144 to 7333586049387978752. Current Snapshot: MvccSnapshot[applied={T|T < 7333586049494278144}]
I20260926 12:38:01.636245 1049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b1d107b715a4bb7b66abee1e3ce605a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43317 } } peers { permanent_uuid: "945659a260fb4f03bfe4fff2d9cd32e8" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } } peers { permanent_uuid: "8b1d107b715a4bb7b66abee1e3ce605a" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } } }
I20260926 12:38:01.636375 1049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b1d107b715a4bb7b66abee1e3ce605a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:38:01.639760 1048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43317 } } peers { permanent_uuid: "945659a260fb4f03bfe4fff2d9cd32e8" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } } peers { permanent_uuid: "8b1d107b715a4bb7b66abee1e3ce605a" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } } }
I20260926 12:38:01.639897 1048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52 [sys.catalog]: This master's current role is: LEADER
I20260926 12:38:01.641001 1098 mvcc.cc:205] Tried to move back new op lower bound from 7333586049494278144 to 7333586049387978752. Current Snapshot: MvccSnapshot[applied={T|T < 7333586049494278144}]
I20260926 12:38:01.641176 1077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 945659a260fb4f03bfe4fff2d9cd32e8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f8aa8d94fe249ffa96d0a36f5387e52" member_type: VOTER last_known_addr { host: "127.10.98.62" port: 43317 } } peers { permanent_uuid: "945659a260fb4f03bfe4fff2d9cd32e8" member_type: VOTER last_known_addr { host: "127.10.98.61" port: 36301 } } peers { permanent_uuid: "8b1d107b715a4bb7b66abee1e3ce605a" member_type: VOTER last_known_addr { host: "127.10.98.60" port: 33279 } } }
I20260926 12:38:01.641284 1077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 945659a260fb4f03bfe4fff2d9cd32e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 12:38:01.641769 1100 mvcc.cc:205] Tried to move back new op lower bound from 7333586049494278144 to 7333586049387978752. Current Snapshot: MvccSnapshot[applied={T|T < 7333586049494278144}]
I20260926 12:38:01.642787 1093 catalog_manager.cc:1347] Generated new cluster ID: 2aca7194012f47d79c16d4d47c2b44c3
I20260926 12:38:01.642853 1093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 12:38:01.677321 1093 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 12:38:01.681461 1093 catalog_manager.cc:1504] Loading token signing keys...
I20260926 12:38:01.698557 1093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f8aa8d94fe249ffa96d0a36f5387e52: Generated new TSK 0
I20260926 12:38:01.699512 1093 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 12:38:01.732385 967 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:38:01.729046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.10.98.61:36301)
0926 12:38:01.729220 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.729226 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.731256 (+ 2030us) server_negotiation.cc:407] Connection header received
0926 12:38:01.731471 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.731475 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.731529 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.731595 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.732242 (+ 647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54022: BlockingRecv error: recv got EOF from 127.0.0.1:54022 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260926 12:38:01.733816 1044 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:38:01.730023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45714 (local address 127.10.98.60:33279)
0926 12:38:01.730173 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.730178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.732916 (+ 2738us) server_negotiation.cc:407] Connection header received
0926 12:38:01.733064 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.733068 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.733142 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.733193 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.733697 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45714: BlockingRecv error: recv got EOF from 127.0.0.1:45714 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 12:38:01.735149 890 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:38:01.730599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36204 (local address 127.10.98.62:43317)
0926 12:38:01.730722 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.730726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.734267 (+ 3541us) server_negotiation.cc:407] Connection header received
0926 12:38:01.734379 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.734382 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.734432 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.734504 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.735028 (+ 524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36204: BlockingRecv error: recv got EOF from 127.0.0.1:36204 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 12:38:01.739681 967 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:38:01.738171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.10.98.61:36301)
0926 12:38:01.738281 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.738286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.738728 (+ 442us) server_negotiation.cc:407] Connection header received
0926 12:38:01.738942 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.738946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.739018 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.739067 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.739561 (+ 494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54034: BlockingRecv error: recv got EOF from 127.0.0.1:54034 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 12:38:01.746421 967 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:38:01.743228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54050 (local address 127.10.98.61:36301)
0926 12:38:01.743334 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.743338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.745433 (+ 2095us) server_negotiation.cc:407] Connection header received
0926 12:38:01.745656 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.745659 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.745714 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.745766 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.746302 (+ 536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54050: BlockingRecv error: recv got EOF from 127.0.0.1:54050 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 12:38:01.752401 967 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:38:01.750748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54064 (local address 127.10.98.61:36301)
0926 12:38:01.750862 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.750866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.751430 (+ 564us) server_negotiation.cc:407] Connection header received
0926 12:38:01.751647 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.751651 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.751708 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.751759 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.752288 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54064: BlockingRecv error: recv got EOF from 127.0.0.1:54064 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 12:38:01.761390 967 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:38:01.759662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54068 (local address 127.10.98.61:36301)
0926 12:38:01.759812 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.759816 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.760423 (+ 607us) server_negotiation.cc:407] Connection header received
0926 12:38:01.760633 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.760637 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.760690 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.760739 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.761274 (+ 535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54068: BlockingRecv error: recv got EOF from 127.0.0.1:54068 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260926 12:38:01.773509 967 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:38:01.771656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54080 (local address 127.10.98.61:36301)
0926 12:38:01.771775 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.771779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.772396 (+ 617us) server_negotiation.cc:407] Connection header received
0926 12:38:01.772636 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.772640 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.772698 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.772755 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.773368 (+ 613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54080: BlockingRecv error: recv got EOF from 127.0.0.1:54080 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 12:38:01.793464 967 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:38:01.791853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54094 (local address 127.10.98.61:36301)
0926 12:38:01.791969 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.791974 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.792615 (+ 641us) server_negotiation.cc:407] Connection header received
0926 12:38:01.792752 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.792756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.792805 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.792871 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.793344 (+ 473us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54094: BlockingRecv error: recv got EOF from 127.0.0.1:54094 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260926 12:38:01.828904 967 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:38:01.827715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54104 (local address 127.10.98.61:36301)
0926 12:38:01.827892 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.827896 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.827968 (+ 72us) server_negotiation.cc:407] Connection header received
0926 12:38:01.828144 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.828147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.828200 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.828279 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.828748 (+ 469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54104: BlockingRecv error: recv got EOF from 127.0.0.1:54104 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
W20260926 12:38:01.896219 967 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:38:01.895136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54116 (local address 127.10.98.61:36301)
0926 12:38:01.895301 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:01.895304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:01.895375 (+ 71us) server_negotiation.cc:407] Connection header received
0926 12:38:01.895503 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:01.895505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:01.895553 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:01.895625 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:01.896101 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54116: BlockingRecv error: recv got EOF from 127.0.0.1:54116 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 12:38:02.027956 967 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:38:02.026632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54128 (local address 127.10.98.61:36301)
0926 12:38:02.026840 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:02.026844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:02.026948 (+ 104us) server_negotiation.cc:407] Connection header received
0926 12:38:02.027202 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:02.027206 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:02.027254 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:02.027325 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:02.027833 (+ 508us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54128: BlockingRecv error: recv got EOF from 127.0.0.1:54128 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260926 12:38:02.287329 967 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 12:38:02.286260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54140 (local address 127.10.98.61:36301)
0926 12:38:02.286385 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:02.286390 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:02.286523 (+ 133us) server_negotiation.cc:407] Connection header received
0926 12:38:02.286623 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:02.286631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:02.286680 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:02.286748 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 12:38:02.287211 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54140: BlockingRecv error: recv got EOF from 127.0.0.1:54140 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 12:38:02.583818 1071 catalog_manager.cc:1259] Loaded cluster ID: 2aca7194012f47d79c16d4d47c2b44c3
I20260926 12:38:02.583906 1071 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8b1d107b715a4bb7b66abee1e3ce605a: loading cluster ID for follower catalog manager: success
I20260926 12:38:02.585960 1071 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8b1d107b715a4bb7b66abee1e3ce605a: acquiring CA information for follower catalog manager: success
I20260926 12:38:02.587242 1071 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8b1d107b715a4bb7b66abee1e3ce605a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 12:38:02.593763 1088 catalog_manager.cc:1259] Loaded cluster ID: 2aca7194012f47d79c16d4d47c2b44c3
I20260926 12:38:02.593849 1088 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 945659a260fb4f03bfe4fff2d9cd32e8: loading cluster ID for follower catalog manager: success
I20260926 12:38:02.595464 1088 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 945659a260fb4f03bfe4fff2d9cd32e8: acquiring CA information for follower catalog manager: success
I20260926 12:38:02.596632 1088 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 945659a260fb4f03bfe4fff2d9cd32e8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 12:38:02.805441 1170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:02.801608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54156 (local address 127.10.98.61:36301)
0926 12:38:02.801841 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:02.801845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:02.801923 (+ 78us) server_negotiation.cc:407] Connection header received
0926 12:38:02.802090 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:02.802092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:02.802215 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:02.802312 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:02.802571 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:02.803116 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:02.803989 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:02.804220 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:02.804306 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:02.804618 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:02.805040 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:02.805194 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:02.805233 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 12:38:02.805289 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":68,"threads_started":1}
I20260926 12:38:02.812003 1176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:02.808404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45726 (local address 127.10.98.60:33279)
0926 12:38:02.808629 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:02.808634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:02.808647 (+ 13us) server_negotiation.cc:407] Connection header received
0926 12:38:02.808761 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:02.808765 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:02.808881 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:02.808983 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:02.809216 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:02.809730 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:02.810533 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:02.810760 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:02.810847 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:02.811142 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:02.811568 (+ 426us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:02.811687 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:02.811783 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 12:38:02.811853 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":60,"threads_started":1}
I20260926 12:38:02.818790 1170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:02.815166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54158 (local address 127.10.98.61:36301)
0926 12:38:02.815314 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:02.815319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:02.815335 (+ 16us) server_negotiation.cc:407] Connection header received
0926 12:38:02.815515 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:02.815517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:02.815561 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:02.815625 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:02.815841 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:02.816551 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:02.817679 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:02.817916 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:02.818016 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:02.818298 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:02.818382 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:02.818442 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:02.818592 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 12:38:02.818648 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 12:38:02.819469 1176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:02.815145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45728 (local address 127.10.98.60:33279)
0926 12:38:02.815506 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:02.815511 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:02.815730 (+ 219us) server_negotiation.cc:407] Connection header received
0926 12:38:02.815898 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:02.815901 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:02.815951 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:02.816019 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:02.816287 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:02.816891 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:02.818066 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:02.818368 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:02.818422 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:02.818899 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:02.818983 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:02.819061 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:02.819325 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 12:38:02.819380 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281}
I20260926 12:38:02.820170 1183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 12:38:02.814939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36212 (local address 127.10.98.62:43317)
0926 12:38:02.815632 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0926 12:38:02.815637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 12:38:02.815652 (+ 15us) server_negotiation.cc:407] Connection header received
0926 12:38:02.815754 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 12:38:02.815758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 12:38:02.815945 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 12:38:02.816057 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 12:38:02.816402 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:02.817154 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:02.818495 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 12:38:02.818810 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 12:38:02.818918 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 12:38:02.819346 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 12:38:02.819717 (+ 371us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 12:38:02.819812 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 12:38:02.819953 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 12:38:02.820014 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":75,"threads_started":1}
I20260926 12:38:02.823088 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 817
I20260926 12:38:02.828647 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 893
I20260926 12:38:02.833875 10632 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskFMA2CE/build/debug/bin/kudu with pid 968
2026-09-26T12:38:02Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3011 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6083 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (285447 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST